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1 File and storage systems: The Google file system 
0 Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung 

v October 2003 Proceedings of the nineteenth ACM symposium on Operating systems 
principles 
Publisher: ACM Press 

Full text available: f| ) pdf(275.54 KB) Additional Information: full citation , references , index terms 



Keywords: clustered storage, data storage, fault tolerance, scalability 



2 Distributed file systems: concepts and examples 
Eliezer Levy, Abraham Silberschatz 

^ December 1990 ACM Computing Surveys (CSUR), volume 22 issue 4 
Publisher: ACM Press 

Full text available: f£) pdf(5.33 MB) Additional Information: full citation , abstract , references , citings , index 

terms , review 

The purpose of a distributed file system (DFS) is to allow users of physically distributed 
computers to share data and storage resources by using a common file system. A typical 
configuration for a DFS is a collection of workstations and mainframes connected by a 
local area network (LAN). A DFS is implemented as part of the operating system of each 
of the connected computers. This paper establishes a viewpoint that emphasizes the 
dispersed structure and decentralization of both data and con ... 

3 A low-bandwidth network file system 

<|b< Athicha i Muthitacharoen, Benjie Chen, David Mazteres 

^ October 2001 ACM SIGOPS Operating Systems Review , Proceedings of the eighteenth 
~ ACM symposium on Operating systems principles SOSP '01, volume 35 issue 

Publisher: ACM Press 

Full text available* pdf(1.29 MB) Additional Information: full citation , abstract , references , otings, index 
™ terms 

Users rarely consider running network file systems over slow or wide-area networks, as 
the performance would be unacceptable and the bandwidth consumption too high. 
Nonetheless, efficient remote file access would often be desirable over such networks— 
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particularly when high latency makes remote login sessions unresponsive. Rather than 
run interactive programs such as editors remotely, users could run the programs locally 
and manipulate remote files through the file system. To do so, however, wo ... 

Minerva: An automated resource provisioning tool for large-scale storage systems 
Guillermo A. Alvarez, Elizabeth Borowsky, Susie Go, Theodore H. Romer, Ralph Becker- 
Szendy, Richard Golding, Arif Merchant, Mirjana Spasojevic, Alistair Veitch, John Wilkes 
November -200-1-ACM Transactions on Computer Systems (TOCS), Volume 19 issue 4 
Publisher: ACM Press 

Full text available- W\ pdf(701 98 KB) Add ' tional Information: full citation , abstract , references , citings , index 
^ terms 

Enterprise-scale storage systems, which can contain hundreds of host computers and 
^storage devices and up to tens of thousands of disks and logical volumes, are difficult to 
design. The volume of choices that need to be made is massive, and many choices have 
unforeseen interactions. Storage system design is tedious and complicated to do by hand, 
usually leading to solutions that are grossly over-provisioned, substantially under- 
performing or, in the worst case, both.To solve the configuration ni ... 

Keywords: Disk array, RAID, automatic design 



; A weighted voting algorithm for replicated directories 

Joshua J. Bloch, Dean S. Daniels, Alfred Z. Spector 
^ October 1987 Journal of the ACM (JACM), volume 34 issue 4 

Publisher: ACM Press 

Full text available- f^ l pdf(4.12 MB) Additional Information: full citation , abstract, references , citings , index 
^ terms , review 

^Weighted voting is used as the basis for a replication technique for directories. This 
technique affords arbitrarily high data availability as well as high concurrency. Efficient 
, algorithms are presented for all of the standard directory operations. A structural property 
^ of the replicated directory that permits the construction of an efficient algorithm for 
^ deletion is proven. Simulation results are presented and the system is modeled and 
analyzed. The analysis agrees well with the simulati ... 

Status report of the graphic standards planning committee of ACM/SIGGRAPH: 

% State-of-the-art of graphic software packages 
Compuater Graphics staff 

September 1977 ACM SIGGRAPH Computer Graphics, Volume n issue 3 
Publisher: ACM Press 

Full text available: pdf(9.03 MB) Additional Information: full citation , references 



7 Recovery in the Calypso file system 
^ Murthy Devarakonda, Bill Kish, Ajay Mohindra 

August 1996 ACM Transactions on Computer Systems (TOCS), volume 14 issue 3 

Publisher: ACM Press 

Full text available: P|pdf(318.88 KB) Additional Information: full citation , abstract, references , dtmgs, index 
LL - H ^ terms , review 

This article presents the deign and implementation of the recovery scheme in Calypso. 
Calypso is a cluster-optimized, distributed file system for UNIX clusters. As in Sprite and 
AFS, Calypso servers are stateful and scale well to a large number of clients. The recovery 
scheme in Calypso is nondisruptive, meaning that open files remain open, client modified 
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data are saved, and in-flight operations are properly handled across server recover. The 
scheme uses distributed state amount the client ... 

Keywords: Calypso, cluster systems, distributed state, state reconstruction 



8 I/O reference behavior of production database workloads and the TPC benchmarks — p ll] 
^ an analysis at the logical level 

Windsor W. Hsu, Alan Jay Smith, Honesty C. Young 

March 2001 ACM Transactions on Database Systems (TODS), volume 26 issue l 
Publisher: ACM Press 

Full text available: *P| p df(5.42 MB) Additional Information: full citation , abstract , references , citings , index 
' ^ terms 

As improvements in processor performance continue to far outpace improvements in 
storage performance, I/O is increasingly the bottleneck in computer systems, especially in 
large database systems that manage huge amoungs of data. The key to achieving good 
I/O performance is to thoroughly understand its characteristics. In this article we present 
a comprehensive analysis of the logical I/O reference behavior of the peak 
productiondatabase workloads from ten of the world's largest corporatio ... 

Keywords: I/O, TPC benchmarks, caching, locality, prefetching, production database 
workloads, reference behavior, sequentiality, workload characterization 



Optimistic replication 
Yasushi Saito, Marc Shapiro 

March 2005 ACM Computing Surveys (CSUR), volume 37 issue l 
Publisher: ACM Press 

Full text available: |£| pdf(656.72 KB) Additional Information: full citation , abstract , references , index terms 

Data replication is a key technology in distributed systems that enables higher availability 
and performance. This article surveys optimistic replication algorithms. They allow replica 
contents to diverge in the short term to support concurrent work practices and tolerate 
failures in low-quality communication links. The importance of such techniques is 
increasing as collaboration through wide-area and mobile networks becomes 
popular.Optimistic replication deploys algorithms not seen in tradition ... 

Keywords: Replication, disconnected operation, distributed systems, large scale systems, 
optimistic techniques 



10 A High Availability Clustering Solution 
Phil Lewis ^ 

August 1999 Linux Journal 

Publisher: Specialized Systems Consultants, Inc. 

Full text available: |g] htmU34.77 KB) Additional Information: full citation , abstract , index terms 

Mr. Lewis tells us how he designed and implemented a simple high-availability solution for 
his company 

11 Pen computing: a technology overview and a vision 
Andr6 Meyer 

>r July 1995 ACM SIGCHI Bulletin, Volume 27 Issue 3 
Publisher: ACM Press 

Full text available: || )pdf(5.14 MB) Additional Information: full citation , abstract , citings , index terms 
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This work gives an overview of a new technology that is attracting growing interest in 
public as well as in the computer industry itself. The visible difference from other 
technologies is in the use of a pen or pencil as the primary means of interaction between 
a user and a machine, picking up the familiar pen and paper interface metaphor. From 
this follows a set of consequences that will be analyzed and put into context with other 
emerging technologies and visions. Starting with a short historic ... 

12 An end-to-end approach to globally scalable network storage 

Micah Beck, Terry Moore, James S. Plank 
▼ August 2002 ACM SIGCOMM Computer Communication Review , Proceedings of the 
2002 conference on Applications, technologies, architectures, and 
protocols for computer communications SIGCOMM '02, volume 32 issue 4 
Publisher: ACM Press 

Full text available* ffil pdf(286 82 KB) Add ' tional Information: full citation , abstract , references , citings , index 

terms 

This paper discusses the application of end-to-end design principles, which are 
characteristic of the architecture of the Internet, to network storage. While putting 
storage into the network fabric may seem to contradict end-to-end arguments, we try to 
show not only that there is no contradiction, but also that adherence to such an approach 
is the key to achieving true scalability of shared network storage. After discussing end-to- 
end arguments with respect to several properties of network stora ... 

Keywords: IBP, asynchronous communications, end-to-end design, exNode, internet 
backplane protocol, logistical networking, network storage, scalability, store and forward 
network, wide area storage 



13 MIL primitives for querying a fragmented world 
Peter A. Boncz, Martin L Kersten 

October 1999 The VLDB Journal — The International Journal on Very Large Data 

Bases, Volume 8 Issue 2 

Publisher: Springer-Verlag New York, Inc. 

Full text available: ^pdf(261.36 KB) Additional Information: full citation , abstract , ci tings , index terms 

In query-intensive database application areas, like decision support and data mining, 
systems that use vertical fragmentation have a significant performance advantage. In 
order to support relational or object oriented applications on top of such a fragmented 
data model, a flexible yet powerful intermediate language is needed. This problem has 
been successfully tackled in Monet, a modern extensible database kernel developed by 
our group. We focus on the design choices made in the Monet interprete ... 

Keywords: Database systems, Main-memory techniques, Query languages, Query 
optimization, Vertical fragmentation 



14 PCD — disk caching disk: a new approach for boosting I/O performance 
Yiming Hu, Qing Yang 

May 1996 ACM SIGARCH Computer Architecture News , Proceedings of the 23rd 

annual international symposium on Computer architecture ISCA '96, volume 

24 Issue 2 

Publisher: ACM Press 

Full text available- ffilpdf(1.29 MB) Additional information: full citation , abstract references , ci tings , index 
^ terms 

This paper presents a novel disk storage architecture called DCD, Disk Caching Disk, for 
the purpose of optimizing I/O performance. The main idea of the DCD is to use a small log 
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disk, referred to as cache-disk, as a secondary disk cache to optimize write performance. 
While the cache-disk and the normal data disk have the same physical properties, the 
access speed of the former differs dramatically from the latter because of different data 
units and different ways in which d ... 

15 Searching the Web | 
jb± August 2001 ACM Transactions on Internet Technology (TOIT), Volume 1 Issue 1 
^ Publisher: ACM Press 

Full text available: f B pdf(319.98 KB) Additional Information: full citation , abstract, references , citings, index 
L - Jh ^ terms , review 

We offer an overview of current Web search engine design. After introducing a generic 
search engine architecture, we examine each engine component in turn. We cover 
crawling, local Web page storage, indexing, and the use of link analysis for boosting 
search performance. The most common design and implementation techniques for each of 
these components are presented. For this presentation we draw from the literature and 
from our own experimental search engine testbed. Emphasis is on introduci ... 

Keywords: HITS, PageRank, authorities, crawling, indexing, information retrieval, link 
analysis, search engine 



16 Special issue: Game-playing programs: theory and practice 
M. A. Bramer 

^ April 1982 ACM SIGART Bulletin, issue 80 

Publisher: ACM Press 

Full text available: pdf(9.23 MB) Additional Information: full citation , abstract 

This collection of articles has been brought together to provide SIGART members with an 
overview of Artificial Intelligence approaches to constructing game-playing programs. 
Papers on both theory and practice are included. 

17 Level II technical support in a distributed computing environment 
Tim Leehane 

September 1996 Proceedings of the 24th annual ACM SIGUCCS conference on User 

services 
Publisher: ACM Press 

Full text available: ^ pdf(5.73 MB) Additional Information: full citation , references , index terms 



18 Designing computer systems with MEMS-based storage 
Jk* Steven W. Schlosser, John Linwood Griffin, David F. Nagle, Gregory R. Ganger 

November 2000 ACM SIGOPS Operating Systems Review , ACM SIGARCH Computer 

Architecture News , Proceedings of the ninth international conference 
on Architectural support for programming languages and operating 

Systems ASPLOS-DC, Volume 34 , 28 Issue 5 , 5 

Publisher: ACM Press 

Full text available: f£ l pdf(439.06 KB) Additional Information: full citation , abstract , references , dtings, index 

terms 

For decades the RAM-to-disk memory hierarchy gap has plagued computer architects. An 
exciting new storage technology based on microelectromechanical systems (MEMS) is 
poised to fill a large portion of this performance gap, significantly reduce system power 
consumption, and enable many new applications. This paper explores the system-level 
implications of integrating MEMS-based storage into the memory hierarchy. Results show 
that standalone MEMS-based storage reduces I/O stall times by 4-74X over ... 
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19 Designing computer systems with MEMS-based storage 

Jb Steven W. Schlosser, John Linwood Griffin, David F. Nagle, Gregory R. Ganger 
^ November 2000 ACM SIGPLAN Notices, volume 35 issue n 
Publisher: ACM Press 

Full text available: ^ pdf(439.06 KB) Additional Information: full citation , abstract , references , index terms 

For decades the RAM-to-disk memory hierarchy gap has plagued computer architects. An 
exciting new storage technology based on microelectromechanical systems (MEMS) is 
poised to fill a large portion of this performance gap, significantly reduce system power 
V s consumption, and enable many new applications. This paper explores the system-level 
^ ^ implications of integrating MEMS-based storage into the memory hierarchy. Results show 
that standalone MEMS-based storage reduces I/O stall times by 4--74X ove ... 

20 Compact deterministic distributed dictionaries (extended abstract) 
M± Kami Gilon, David Peleg 

v July 1991 Proceedings of the tenth annual ACM symposium on Principles of 
distributed computing 
Publisher: ACM Press 

Full text available: 11 3 pdf(1.29 MB) Additional Information: full citation , references , index terms 
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21 An end-to-end approach to globally scalable programmable networking 
A Micah Beck, Terry Moore, James S. Plank 

>^ August 2003 ACM SIGCOMM Computer Communication Review , Proceedings of the 
ACM SIGCOMM workshop on Future directions in network architecture 

FDNA '03, Volume 33 Issue 4 

Publisher: ACM Press 

Full text available: ^| pdf(447.96 KB) Additional Information: full citation , abstract , references , index terms 

The three fundamental resources underlying Information Technology are bandwidth, 
storage, and computation. The goal of wide area infrastructure is to provision these 
resources to enable applications within a community. The end-to-end principles provide a 
scalable approach to the architecture of the shared services on which these applications 
depend. As a prime example, IP and the Internet resulted from the application of these 
principles to bandwidth resources. A similar application to storage r ... 

Keywords: Internet Backplane Protocol, Logistical Network Computing, active 
networking, asynchronous communications, distributed state management, end-to-end 
design, network storage, programmable networking, scalability, store and forward 
network 



22 The TickerTAlP parallel RAID architecture 

Pei Cao, Swee Boon Lin, Shivakumar Venkataraman, John Wilkes 
August 1994 ACM Transactions on Computer Systems (TOCS), volume 12 issue 3 
Publisher: ACM Press 

Full text available: W\ pdf(2.04 MB) Additional Information: full citation , abstract , references , citings , index 
' ^ terms 

Traditional disk arrays have a centralized architecture, with a single controller through 
which all requests flow. Such a controller is a single point of failure, and its performance 
limits the maximum number of disks to which the array can scale. We describe 
TickerTAlP, a parallel architecture for disk arrays that distributes the controller functions 
across several loosely coupled processors. The result is better scalability, fault tolerance, 
and flexibility.This article present ... 

Keywords: RAID disk array, decentralized parity calculation, disk scheduling, distributed 
controller, fault tolerance, parallel controller, performance simulation 
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23 Lifestreams: a storage model for personal data 
^ Eric Freeman, David Gelernter 

^ March 1996 ACM SIGMOD Record, Volume 25 issue l 
Publisher: ACM Press 

Full text available: | ||pdf(691.07 KB) Additional Information: full citation , abstract, a tings 

Conventional software systems, such as those based on the "desktop metaphor," are ill- 
equipped to manage the electronic information and events of the typical computer user. 
We introduce a new metaphor, Lifestreams, for dynamically organizing a user's personal 
workspace. Lifestreams uses a simple organizational metaphor, a time-ordered stream of 
documents, as an underlying storage system. Stream filters are used to organize, monitor 
and summarize informat ... 

24 Performing remote operations efficiently on a local computer network 
^ Alfred Z. Spector 

April 1982 Communications of the ACM, Volume 25 issue 4 
Publisher: ACM Press 

Full text available: f3 pdf(1.58 MB) Additional Information: full citation , abstract, references , dtmgs, index 
^ terms 

A communication model is described that can serve as a basis for a highly efficient 
communication subsystem for local networks. The model contains a taxonomy of 
communication instructions that can be implemented efficiently and can be a good basis 
for interprocessor communication. These communication instructions, called remote 
references, cause an operation to be performed by a remote process and, optionally, 
cause a value to be returned. This paper also presents implementation considerati ... 

Keywords: communication models, efficient communication, transactions 



An Introduction To DIAM Levels Of Abstraction In Accessing Information 
Michael Levin 

December 1978 Proceedings of the 1978 annual conference 
Publisher: ACM Press 

Full text available: pdf(104 MB) Additional Information: full citation , abstract , references , index terms 

There has been a great deal of interest in information models that are independent of all 
considerations of implementation. But such models cannot be exploited to their fullest 
potential without a theory for modeling implementations and the mapping of information 
models into implementations. DIAM (Data Independent Access Model) is such a theory, 
describing implementations on several levels of abstraction and the mappings from level 
to level. This paper is a tutorial which provides the reade ... 

Keywords: DIAM, Data independence, Data model, Database, Information management, 
Relational DIAM, Relational model 




26 ObjectGlobe: Ubiquitous query processing on the Internet 

R. Braumandl, M. Keidl, A. Kemper, D. Kossmann, A. Kreutz, S. Seltzsam, K. Stocker 
August 2001 The VLDB Journal — The International Journal on Very Large Data 

Bases, Volume 10 Issue 1 

Publisher: Springer-Verlag New York, Inc. 

Full text available: ^g ) pdf(251.44 KB) Additional Information: full citation , abstract , citings , index terms 

We present the design of ObjectGlobe, a distributed and open query processor for 
Internet data sources. Today, data is published on the Internet via Web servers which 
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have, if at all, very localized query processing capabilities. The goal of the ObjectGlobe 
project is to establish an open marketplace in which data and query processing 
capabilities can be distributed and used by any kind of Internet application. Furthermore, 
ObjectGlobe integrates cycle providers (i.e., machi ... 

Keywords: Cycle-, function- and data provider, Distributed query processing, Open 
systems, Privacy, Quality of service, Query optimization, Security 



27 National id card: the next generation: The US/Mexico border crossing card (BCC): a 




|k case study in biometric, machine-readable id 
^ Andrew Schulman 

April 2002 Proceedings of the 12th annual conference on Computers, freedom and 

privacy 
Publisher: ACM Press 

Full text available: W\ htm(187.31 KB) Additional Information: full citation , index terms 



28 Special issue on knowledge representation Q 
J*^ Ronald J. Brachman, Brian C. Smith 

February 1980 ACM SIGART Bulletin, issue 70 

Publisher: ACM Press 

Full text available: || |pdfn3.13 MB) Additional Information: full citation , abstract 

In the fall of 1978 we decided to produce a special issue of the SIGART Newsletter 
devoted to a survey of current knowledge representation research. We felt that there 
were twe useful functions such an issue could serve. First, we hoped to elicit a clear 
picture of how people working in this subdiscipline understand knowledge representation 
research, to illuminate the issues on which current research is focused, and to catalogue 
what approaches and techniques are currently being developed. Secon ... 

29 A high-availability high-performance e-mail cluster Q 
Wyman Miles 

^ November 2002 Proceedings of the 30th annual ACM SIGUCCS conference on User 
services 
Publisher: ACM Press 

Full text available: f|l pdf(175.10 KB) Additional Information: full citation , index terms 



Keywords: anti-spam, anti-virus, cluster, electronic mail, failover, high-availability, high- 
performance, mail routing, proxy, redundancy 



30 A program for the analysis of speech error data 
Georgette Silva, Victoria A. Fromkin, Frank Heath 
December 1976 ACM SIGLASH Newsletter, Volume. 10 issue 1-2 
Publisher: ACM Press 

Full text available: ^| pdf(1.21 MB) Additional Information: full citation , abstract , references , citings 

The first extensive investigation of speech errors, i.e., utterances which in some way 
deviate from the intended or target utterances, is probably that of Rudolf Meringer (1,2), 
whose published collection of over eight thousand German speech, reading, hearing, and 
writing errors has been a rich source of data for several subsequent generations of 
researchers. Over the years, speech error data have come to be regarded as a kind of 
"window" into linguistic mental processes which can provide the in ... 



http://portal.acm.org/results.cfm?query=%2Bfirst%20%2Bsecondai7% 11/16/05 



Results (page 2): +first +secondary +mirror +file +systems +temporary +storage +delete 



Page 4 of 5 



31 The TickerTAlP parallel RAID architecture 

Pei Cao, Swee Boon Lim, Shivakumar Venkataraman, John Wilkes 
>^ May 1993 ACM SIGARCH Computer Architecture News , Proceedings of the 20th 

annual international symposium on Computer architecture ISCA '93, volume 

21 Issue 2 

Publisher: ACM Press 

Full text available: f£ |pdf(1.19 MB) Additional Information: full citation , abstract , references , citings, index 
^ terms 

Traditional disk arrays have a centralized architecture, with a single controller through 
which all requests flow. Such a controller is a single point of failure, and its performance 
limits the maximum size that the array can grow to. We describe here TickerTAlP, a 
parallel architecture for disk arrays that distributed the controller functions across several 
loosely-coupled processors. The result is better scalability, fault tolerance, and flexibility. 
This paper presents the Tic ... 

32 A probe-based monitoring scheme for an object-oriented distributed operating system 
Partha Dasgupta 

June 1986 ACM SIGPLAN Notices , Conference proceedings on Object-oriented 

programming systems, languages and applications OOPLSA '86 r volume 21 

Issue 11 

Publisher: ACM Press 

Full text available: 1|| pdf(762.64 KB) Additional Information: full citation , references , citings , index terms 



33 DBMS implementation experience: An entitv-based database user interface 
R. G. G. Cattell 

^ May 1980 Proceedings of the 1980 ACM SIGMOD international conference on 
Management of data 

Publisher: ACM Press 

Full text available: ^pdf(815.00 KB) Additional Information: full citation , abstract , references , a tings 

A user interface to a database designed for casual, interactive use is presented. The 
interface is entity-based: the data display to the user is based upon entities (e.g., 
persons, documents, organizations) that participate in relationships, rather than upon 
relations alone as in the relational data model. Examples from an implementation of the 
system are shown, for a prototype personal database (PDB), developed in connection with 
the ZOG system at Carnegie-Mellon University (Robertson et ... 

34 Why web-based network monitoring? Leveraging the platform 
Ron D. Jenkins 

May 1999 International Journal of Network Management, volume 9 issue 3 
Publisher: John Wiley & Sons, Inc. 

Full text available: pdf(494.08 KB) Additional Information: full citation , abstract index terms 

The increasing use of network monitoring and the growth of the Internet and intranets are 
converging trends that make IP network infrastructures the logical means of delivering 
network monitoring, using browser -bused clients. Copyright © 1999 John Wiley & Sons, 
Ltd. 
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Data Storage Management 

The backup window cannot vary with the amount of file system activity, ... Rather than 
simply make a copy of the files on the secondary storage, the files ... 
www.nss.co.uk/VVhitePapers/ data_storage_mmenVdata_storage_mment.htm - 298k - 
Cached - Similar pages 

[pdf] VERITAS Software Corporation 

File Format: PDF/Adobe Acrobat - View as HTML 

retrieve it from the secondary storage system. The type dependent file system, ... In 
order to calculate the disk address, the type dependent file system ... 
eval.veritas.com/webfiles/docs/fswht1 .pdf - Similar pages 

[pdf] Faster. Safer Oracle Data Migration 
File Format: PDF/Adobe Acrobat - View as HTML 

file system, the data is consistent if the file system check utility can be run ... build the 
secondary storage and re-initialize using one of the methods ... 
eval.veritas.com/mktginfo/products/ 

White_Papers/Storage_Server_Management/sf_vvr_wp.pdf - Similar pages 
[ More results from eval.veritas.com ] 

cpdf] Replication Means Different Things to Different People 
File Format: PDF/Adobe Acrobat - View as HTML 

written to the primary disk, asynchronous replication is not bound by the ... different, even 
with different operating systems. * Distribute files from a ... 
www.signiant.com/documents/WP_FormsOfReplication.pdf - Similar pages 

[PDF] Storage Management for Object-Oriented Database Management Systems ... 
File Format: PDF/Adobe Acrobat - View as HTML 

File Objects. In the EXODUS storage system, file objects are collections (sets) of ... Also, 
objects within a given file object are placed only on the disk ... 
www.cs.uwaterloo.ca/research/tr/1991/46/storaqe.pdf- Similar pages 

epdf] Sun StorEdge Availability Suite 3.1 Remote Mirror Software ... 
File Format: PDF/Adobe Acrobat - View as HTML 

The remote mirror software is not a file or file system replicator. ... The remote mirror 
software on the primary system (hostl) requests disk blocks ... 
www.sun.com/products-n-solutions/ hardware/docs/pdf/81 6-441 5-10.pdf- Similar pages 

New college graduate jobs 

It's a perfect test disk which will challenge your system, from CD-ROM to ... Each write 
transaction to the disk is sent to the remote mirror disk and the ... 
www.baoshan-yunnan.com.cn/jobs/articles/ new-college-graduate-jobs/new-college- 
graduate-jobs.html - 49k - Cached - Similar pages 

Hosting mysql php server 

Each write transaction to the disk is sent to the remote mirror disk and the ... The most 
commonly used file system protocols (Windows CIFS and Unix NFS) ... 
www.baoshan-yunnan.com.cn/computing/articles/ hosting-mysql-php-server/hosting-mysql- 
php-server.html - 69k - Cached - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

operating Environment and Related Disk Storage systems. ... All mounted File systems 
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the Remote Mirror Software is Not A File or File system Replicator. ... the Remote Mirror 
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Supplemental Result - Similar pages 
[ More results from www.cobaltnet.com ] 

[ppFi FILE: Logical Storage Unit. A set of physical blocks. ACCESS ... 
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... Note, logical block number is translated to a physical block address by the file- 
organization module ... Shadow (Mirror) RAID ... eg, ZIP-disk, diskette, ...) Tapes ... 
hive.cs.uwyo.edu/-yu/COSC4740-01/Lecture1 0.pdf - Supplemental Result - Similar pages 

[pdf] VERITAS NetBackup Release 4.5 Technical Overview 
File Format: PDF/Adobe Acrobat - View as HTML 

the NetBackup Advanced Client and Online File System Backups. ... in most cases, A Third 
Mirror of the Data is created within A Disk Array to serve as the ... 
www.eandm.co.il/events/ veritas/nbu_6_tech_overview_wp.pdf - Supplemental Result - 
Similar pages 
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(Disk-level Replication). Disk. Mirror/SnapCopy. DBMS. Edit loads. A BCDEFGH 
SELECTEDON-LINE ... The log files are temporary locations for information about ... 
www.esri.com/systemsint/kbase/docs/ arcgis_enterprise_geodatabase_replication.pdf- 
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File Format: PDF/Adobe Acrobat - View as HTML 

File-system data snapshots. • Backup to disk. • Data replication ... command to copy data 
from the primary to the secondary storage subsystem. During ... 
www.datacenterjournaLcom/Tools/WhitePapers/ 
Data_Recovery_Choosing_the_Right_Technologies.pdf - Similar pages 
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The file level: refers to the placement of files in disk or tape resources ... We also address 
parallel access from disk and tape systems in other tasks. ... 

sdm.lbl.gov/sdmcenter/detailedtasks.htm - 270k - Nov 14, 2005 - Cached - Similar pages 
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VERITAS Volume Manager can synchronously mirror data natively over storage ... 
database may also use data stored in VERITAS File Systems, created in these ... 
www.dlt.com/storage/PDF/whitepapers/ veritas/ha/vvr white paper.pdf- Similar pages 
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replication tool used to automatically mirror file replicas ... Management System, tape or 
disk based, so that mass storage of files can be ... 
aszt.inf.elte.hu/-grid/thes/lorincz-thesis.pdf- Nov 14, 2005 - Similar pages 

Datacenter Discussion , Review & Ratings - Disaster Recovery and „■ 

... loss on the primary disk, the data is retrieved seamlessly from the mirror disk. ... By 
eliminating the need to deploy matching disk arrays, file servers or ... 
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Method and apparatus for performing point in time backup operation ... 
A backup controller and device connect to the secondary storage device ... During normal 
operations the second system operates to mirror the data from the ... 
www.freepatentsonline.com/5857208.html - 68k - Cached - Similar pages 

Remotely mirrored data storage system with a count indicative of ... 
The secondary storage system controller 44 also includes at least a first ... When the 
primary data storage system controller's disk adapter writes the data ... 
www.freepatentsonline.com/6052797.html - 253k - Cached - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

... and spend innumerable hours of a valuable system administrator's ... this software uses 
a caching area on disk to capture ... that have any potential for files to be ... 
www.dataIink.com/whitepapers/Choosing_ Data_Protection_Technologies_DATALINK.pdf- 
Supplemental Result - Similar pages 
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www.thomsongrassvalley.com/ wp/Tendolkar/gvgprofileMIB.pdf - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

DOCUMENTATION ON THE COMPACT DISC ARE SUBJECT TO THE LICENSE 
AGREEMENTS ... The Virtual File System (VFS) Manager provides a unified API that ... 
www.palmos.com/dev/support/ docs/conduits/win/C++SvncCompanion.pdf - Similar pages 

pdf] COM Sync Suite Companion 

File Format: PDF/Adobe Acrobat - View as HTML 

IF THIS TECHNICAL DOCUMENTATION IS PROVIDED ON A COMPACT DISC, THE 
SOFTWARE AND OTHER ... Expansion Manager and Virtual File System (VFS) Manager 
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www.palmos.com/dev/support/ docs/conduits/win/COMSyncCompanion.pdf - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

... Optimized ^synchronization Resynchronize volumes following disk, link, system, and 
storage ... Version 3.0 does not require the use of an .rhosts file. ... 
www.cobaltmicro.com/products-n-solutions/ hardware/docs/pdf/806-751 2-10.pdf - 
Supplemental Result - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

... s "About Replicating File Systems" on page 4 ... changes that occurred to the original 
disk that "failed ... To transport data, the remote mirror software uses ... 
www.cobaltmicro.com/products-n-solutions/ hardware/docs/pdf/816-441 5-10.pdf- 
Supplemental Result - Similar pages 
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CX600 Disk-Array Storage Systems and offers useful Background ... and 132 G bytes of 
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support.dell.com/support/ edocs/stor-sys/common/01 40031 13.pdf- Supplemental Result - 
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fppR EMC Fibre Channel Storage System Model FC4700-2 Configuration ... 
File Format: PDF/Adobe Acrobat - View as HTML 

operating System must mirror the Images, and has no offsetting ... its Disk Space for user 
files, but does not provide any High Availability. Features. ... 
support.dell.com/support/ edocs/stor-sys/fc47002/cfgplan.pdf - Supplemental Result - 
Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

files are stored consecutively on the secondary Storage Media ... must be augmented by 
the application because the File System Disk Controller is not ware ... 
ftp://ftp.kom.e-technik.tu-darmstadt.de/ pub/papers/GS98-3-paper.pdf - Supplemental 
Result - Similar pages 

[pdf) Development Considerations for Storage Applications in Windows 2000 
File Format: PDF/Adobe Acrobat - View as HTML 

NTFS— The newest version of NTFS has a new on-Disk File System structure ... mirror 
removed from a mirror set again without requiring System reboot. ... 
www.securedatatechnology.com/library/StorDev.pdf-Supplemental Result - Similar pages 

[pdf] Using Snapshot-Enhanced Mirroring to Provide Cost-Effective 
File Format: PDF/Adobe Acrobat - View as HTML 

not until both devices have committed the write to disk that the system ... in the event of a 
temporary communication link disruption, a Snapshot is added ... 
www.basis.biz/.../basis/dwn/SVM-Snapshot-Enhanced-Mirroring. pdf? 
PHPSESSID=3565ab6419b4d541957382845b7c6017 - Supplemental Result - 
Similar pages 

[pdf] Storage Management for SAP and Oracle: Split Mirror Backup ... 
File Format: PDF/Adobe Acrobat - View as HTML 

The file systems requirements follow standard SAP installation, procedures. ... consistent 
state based on the disk image and the online redo log files. ... 
www-03.ibm.com/servers/storage/ disk/ess/pdf/splitmirror-oracle.pdf - Similar pages 

rPDR FFCC ® © MMMMIIBBMM ® ® TTSS ® ® ™ ™ DDSS 44550000 
SSAAPP ... 

File Format: PDF/Adobe Acrobat - View as HTML 

Suspended I/O prevents disk writes while the split mirror image of a database is ... 
Unmount the filesystems that will be mirrored, e. Create the mirror ... 
www-03.ibm.com/servers/storage/ solutions/sap/pdf/metromirror-ds4500.pdf - 
Similar pages 
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Design and Architecture for Database Solutions in Future ... 

In a database system, data and hardware resource conflicts are an important source of ... 

since they need to be written into a temporary disk file. ... 

www.cs.helsinki.fi/research/rodain/barit97/rodain.html - 52k - Cached - Similar pages 
fppR THESIS 

File Format: PDF/Adobe Acrobat - View as HTML 

disk would degrade the system performance because of the high I/O latency. The ... and 
mirror pages can greatly reduce waste bandwidth, disk spaces, ... 
anres.cpe.ku.ac.th/pub/thesis-spider.pdf - Similar pages 
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... server that serves files using a network protocol such as Network File System (NFS) 
or ... Initially, disk and tape storage devices had no on-board intelligence. ... 
www.ece.mtu.edu/-ejng/ Storage%20Network/lntro%20to%20SAN.pdf - Supplemental 
Result - Similar pages 

[PDF] Cost-Effective Disaster Recovery With Snapshot-Enhanced. Any-to „■ 
File Format: PDF/Adobe Acrobat - View as HTML 

... 2. Disk space for snapshots does not need to be ... volumes, such as the data and log 
files in a ...than proprietary mirroring solutions that only mirror data between ... 
www.itstorage.co.uk/Resources/WhitePapers/ files/Cost-Effective%20Disaster% 
20Recovery.pdf - Supplemental Result - Similar pages 

SGI TPL View (tp9500) 

Remote Volume Mirroring - Rejects mirror relationship if primary volume contains ... During 
the download process, the Secondary storage array automatically ... 
techpubs.sgi.com/library/tpl/cgi-bin/getdoc. cgi? 

coll=0650&db=relnotes&fname=/usr/relnotes/tp9500 - 253k - Cached - Similar pages 

SGI TPL (IRIX 6.5: End-User/TPSconcepts - Chapter 3. Remote Volume ... 
Stop all I/O and unmount Any File systems on This Volume before proceeding ... Backups 
to tape or disk can be made while I/O continues Using the Snapcopy ... 
techpubs.sgi.com/library/tpI/cgi-bin/getdoc.cgi/ 

0650/bks/SGI_EndUser/books/TPSconcepts/sgi_html/ch03.html - 231 k - Supplemental 

Result - Cached - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

copying data from a tape archival system in one organisation to a disk drive ... via a file 
cache which is an area on the secondary storage where the remote ... 
arxiv.org/pdf/cs.DC/0506034 - Similar pages 

rppFi Migrating from x86 to PowerPC. Part 2: Anatomy of the Linux boot ... 
File Format: PDF/Adobe Acrobat - View as HTML 

coerce all your other software to put its temporary files in special locations. ... loads the 
operating system kernel directly off disk (or some other ... 

www.developers.net/filestore2/download/875 - Supplemental Result - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 
Supplemental Result - Similar pages 

rppFi arXivxs. DC/0506034 v1 10 Jun 2005 
File Format: PDF/Adobe Acrobat - View as HTML 

copying Data from A tape archival system In one organisation to A disk drive In ... In P2P 
networks, Oceanstore [47] is A Distributed File system that ... 
xxx.sf.nchc.org.tw/pdf/cs.DC/0506034 - Supplemental Result - Similar pages 

[pdf] Choosing Data Protection Technologies 
File Format: PDF/Adobe Acrobat - View as HTML 

... data from the primary to the secondary storage subsystem without ... as a SAN, many 
different file systems, volume management and disk management formats ... 
securitytechnet.com/resource/rsc-center/vendor-wp/ 

datalink/Choosing_Data_Protection_Technologies_DATALINK.pdf - Supplemental Result - 
Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

copying data from a tape archival system in one organisation to a disk drive in ... In P2P 
networks, Oceanstore [47] is a distributed file system that ... 
www.gridbus.org/reports/DataGridTaxonomy.pdf - Similar pages 

[pdf] The Definitive Guide to Windows 2000 Storage Resource Management 
File Format: PDF/Adobe Acrobat 

... You can add disks and extend or mirror a volume ... you can use wizards or the Disk 
Management console to ... used to, such as the Distributed file system (Dfs), which ... 
www.esm-software.nl/documentation/ Storagecentral/E-book%20%20SC5%20VRTS.pdf - 
Supplemental Result - Similar pages 

Encompass. An HP User Group 

This is a mirror of the Gopher client and Lynx WWW-browser available from Foteos ... 
There is no limit to how many files may be so marked on a disk. ... 
www.encompassus.org/ftplib/ LIB_VS0193_ISO9660_AAAREADME_96B.html - 188k - 
Cached - Similar pages 
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COM procedure as executive-mode logicals in the system table; ... does Many disk and 
file maintenance DFU_FREEWARE_024.A;1 ...operations. EMON021-3. ... 
www.encompassus.org/ftplib/ VS0193_VAX000_AAAREADME_96B.html - 196k - 
Supplemental Result - Cached - Similar pages 

Datacenter Discussion , Review & Ratings Forums - Disaster ... 

Mirroring provides the first level of data protection with a mirror disk ... by eliminating the 

need To deploy matching disk arrays, file servers or ... 

www.datacenterchat.com/showthread. php?goto=lastpost&t=1 1 38 - 60k - Supplemental 
Result - Cached - Similar pages 

Migrating from x86 to PowerPC. Part 2: Anatomy of the Linux boot „. 
the disk needs to contain drivers for many different hardware types, ... compressed 
filesystems also have other annoyances, such as the inability to report ... 
www-106.ibm.com/developerworks/ library/pa-migrate2/?ca=dgr-lnxw01BootProcess - 45k - 
Supplemental Result - Cached - Similar pages 

United States Patent Application: 0030046260 

... Operating in parallel and as a mirror image other ... browse its files and folders on their 
hard disk. ... names, sizes, hashes, owners and other file system meta-data ... 
appfM .uspto.gov/netacgi/nph-Parser?Sect1 =PT02& Sect2=HITOFF&p=1 &u=% 
2Fnetahtml%2FPTO%2Fsearch-... - 1 01 k - Supplemental Result - Cached - Similar pages 

United States Patent Application: 0020036694 

... the system obtains A temporary "IP" (Internet Protocol) Address from the local ... [0083] 

the lecture data source files stored on the secondary storage ... 

appfM . uspto.gov/.. ./ 20020036694&RS=DN/20020036694 - 107k - Supplemental Result - 

Cached - Similar pages 
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The disk needs to contain drivers for many different hardware types, ... Compressed 
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Nov 14, 2005 - Cached - Similar pages 
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... that ON - B ar uses to track data objects and the components of the backup and restore 
system, s Appendix A describes the ON - B ar message file and presents ... 
www.ifis.mu-luebeck.de/lehre/ws0001 / dbs/informix- 

doc/answers/pubs/pdf/81 1xpsu/3701a.pdf - Supplemental Result - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 
Supplemental Result - Similar pages 

AIX 3 Announcement Letter. ... 

... automatically maintain multiple copies of (MIRROR) information at ... delays associated 
with the disk operations o Mapped files, which associates file data with ... 
www.dogs-chod.co.uk/howto/3ann.txt - 101 k - Supplemental Result - 
Cached - Similar pages 

From cbentley@beech.advance.com Mon Dec 11 20:35:54 1995 Received ... 
... Gain The following error is reported against Disk Data Input ... output file, which is 
plotted as a mirror image. ... when attempting to write the output file to the ... 
128.253.204.94/geology/promax_bb/known_problems/ 

ProMAX_V6.0_Known_Problems_Nobember_1995 - 79k - Supplemental Result - 
Cached - Similar pages 
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ProMAX_V6.0_Known_Problems_February_1996 - 90k - Supplemental Result - 
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[pdf] The design and implementation of a relational database management ... 
File Format: PDF/Adobe Acrobat - View as HTML 

... Drawing upon the examples provided in academic and commercial relational database 
management systems, a system referred to as LEAP, is proposed, implemented ... 
www.leyton.org/download/leap-design-1 .1 .pdf - Supplemental Result - Similar pages 
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Hard Disk Drive as the Primary Back-up: in duplicated (Mirror mode) ... The system must 
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www.bttb.gov.bd/home/main/download/ITXSpecBK2.doc - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 
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epdf] C:\Documents and Settings\Gail Elias\Scott\Copv of Program ... 
File Format: PDF/Adobe Acrobat - View as HTML 
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insolution.co.kr/Pdf/edm_pdg.pdf - Supplemental Result - Similar pages 
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File Format: PDF/Adobe Acrobat - View as HTML 

management systems, a system referred to as LEAP, is proposed, implemented and. 
tested. ... 5.7.8 Report File, with a Nested expression. ... 
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www.fms.treas.gov/rebids/ lockbox/attachments/attachment_c.pdf- Supplemental Result - 
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Abstract 

Recent new products such as SAP's B2B Procurement, CRM and mySAP.com Portals and 
Trading Exchange markets require an ever-increasing need for continuous system availability. 

This paper provides information on an essential component of advanced infrastructure solutions - 
the High Availability Split Mirror Backup / Recovery (SMBR) for SAP R/3 on the Oracle 
RDBMS and the AIX operating system environments. 

The solution described in this paper is intended to deliver a backup with no impact on live R/3 
system ("serverless") using the advanced functions of IBM's Enterprise Storage Server (ESS). 
This "zero" downtime for the live R/3 system means that SAP users typically do not miss a beat 
while the backup of the live database takes place. No transactions typically are cancelled during 
the copy process / backup process. 

"Instant" availability of a point-in-time copy of the production database using Oracle's HOT 
BACKUP feature provides the ability to deliver a consistent copy of the database using the log 
information written during the online backup. The ability to provide consistent copies of the 
database provides flexibility to place an emergency system at the user's disposal while recovering 
the live database from a disaster. Beyond Backup / Recovery, a consistent copy of the database 
may be used for various purposes, such as creation of Reporting, Production-Fix or a Repository 
instance for a Business Warehouse (BW) system. 
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Preamble 

This white paper, written from the DBA's perspective, addresses the infrastructure design, 
implementation tasks and techniques required for complex Enterprise Application Integration 
landscapes for high availability SAP R/3 applications consisting of a database (Oracle), an 
operating system (AIX) and a Enterprise Storage Subsystem (ESS) which all interoperate to 
deliver an easy-to-manage backup & recovery solution for SAP customers. Backup & Recovery 
solutions are mission critical activities in today's world of 7 x 24 computing and are a major 
focus for IT personnel, for application management and for DBA's. With the exploding growth in 
storage requirements for SAP application environments, this work touches on major elements of 
each area of technology spanning critical operating requirements and how this storage-centric 
solution delivers compelling value for SAP customers. 



1 Introduction 

Service level agreements increasingly have to reflect that in case of planned downtime such as 
database Data Manipulation Language (DML) error, backup, hardware / software maintenance, 
R/3 upgrade and unplanned downtime such as error analysis and restores, the system has to be 
available within minutes. 

SAP's Advanced Technology Group has developed scenarios using live databases that constantly 
copy or mirror using storage subsystems, allowing business continuation during the split (and 
^synchronization) of the mirror. Once the logical database mirrors are established, additional 
copies are created for backup and for use by a standby SAP R/3 System. This solution minimizes 
the I/O load impact of the live environment and offloads the backup activity away from the live 
database / storage server to a standby / backup server host. The Split Mirror solution, based on 
this concept was successfully implemented using Oracle database on the AIX platform using the 
ESS. 
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This solution can be implemented for with a single or a dual ESS configuration using the ESS's 
advanced functions - the local copy function FlashCopy (FC) and the remote synchronous copy 
function Peer-to-Pee-Remote-Copy (PPRC). The core R/3 system was loaded using 
SAP - developed SSQJ tool [12] for OLTP volumes. 

This solution in a dual ESS configuration is intended to enable customers to implement remote 
data vaulting and/or to scale to a larger database size. 

A solution similar to the one described in the following pages was implemented on IBM's ESS 
with SAP R/3 on DB2OS390 and first demonstrated in November 1999 (see references [4, 5] for 
details). Later, a similar solution was validated on the ESS using DB2 UDB on the AIX platform 
[11]. Both these Split Mirror Backup / Recovery solutions on DB2OS390 and on DB2 UDB / 
AIX used a special "write Suspend / Resume" function created by IBM initially for DB2OS390 
and later for DB2 UDB especially for executing the SAP Split Mirror Backup / Recovery 
Solution (SMBR). This open systems solution for SAP with Oracle on AIX takes advantage of 
Oracle's Online or Hot Backup functionality. 

2 Customer Requirements 

Increasingly, with the rapid trend towards very large databases, accompanied by the need for high 
availability in a global computing environment, customers now demand that production systems 
be available on a 7 x 24 basis. This also means that in case of disaster, the system has to be 
available within minutes or hardly longer than the time needed for the physical reload of the 
database from secondary or remote storage media. This high availability requirement also implies 
that backup and the creation of an Emergency system may not cause any downtime of the live 
production system and all procedures to achieve this must be seamlessly automated. 

Customers are also very aware of the fact that software or application logical errors and not 
hardware failures are the most likely causes for the need for disaster recovery capabilities. Hence 
for mission critical applications, customers will do everything possible to optimally protect them 
from a hardware disaster. This means that cost effective, intelligent fault tolerant storage 
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subsystems are an important ingredient of high availability advanced infrastructure solutions, 
helping to insure against cost-prohibitive downtime possibilities. 

Along with the demanding requirements of non-stop, web-centric computing, customers now 
have to contend with business processes spread over multi-vendor platforms, databases and file 
systems, where automated interaction between systems provide the essence of competitive 
productivity. In these emerging environments, the customers, need and demand database 
availability with current data, fast backup / restore / recovery with no impact on the main 
production (OLTP) system all enabled with seamless automation through well defined, user 
friendly management interfaces. With the emerging world of SAP technologies, customer 
environments have to meet stringent requirements for availability, scalability and flexibility that 
can handle changing customer requirements based upon SAP instance strategies, data migration 
requirements and disparate growth rates of databases. 



3 SAP Requirements 

In order to deliver a solution that matches the complex requirements for high availability backup, 
recovery and performance, the SAP workload needs to be taken off the live production system 
and all administrative tasks performed on a copy of the system. The recommended environment is 
depicted in Figure 1 . 
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Figure 1: SAP High Availability Advanced Infrastructure 
Split Mirror Backup & Recovery Environment 

While a production server is connected to a primary fault tolerant storage controller, a mirrored 
remote copy of the production database is created without the computing support of the 
production server, in a separate / secondary fault tolerant storage controller located at a spatially 
separated site, providing high availability and disaster recovery capability for business 
continuance. This copy can be accessed by a standby / backup server should IT management 
procedures require its intervention in the event the primary site experiences business interruption. 
A local copy function within the secondary storage controller produces a consistent point-in-time 
copy of the production database. This consistent copy, at the option of the user, can be either held 
inside secondary storage controller or be transferred to tape for remote vaulting. In the event that 
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the primary database or its mirror is not available, the production server or the backup server can 
be connected to this disk resident copy, helping to dramatically reducing downtime. 

The database and SAP Basis administration can be augmented by providing a homogeneous split 
mirror-based system copy for the purposes of test upgrades, hot packs and database recovery 
routines. Intelligent storage controllers need to deliver this capability to create near-instant, 
consistent copies of the database. 

Backup & restore time is database size independent 
Minimum impact on production environment 
Phyical disaster prevention 
Enhanced database administration 
Remote data vaulting 
- Offloading backup from production database server 
Reduction of backup-restore-time to minutes 

In order to realize this concept, the database management system needs to cooperate with storage 
subsystems to deliver the results. It should support the creation of a consistent database copy 
during the application READ/WRITE processing in a manner that exerts no impact on the 
production system, database or the live production storage server. 

4 Oracle and ESS Features to Support the Split Mirror Backup / 
Recovery Solution 

This section describes the key Oracle features required to create a consistent backup database 
image of the production database using Oracle's Online Hot Backup commands. The backup 
image includes SAP R/3 objects and File definitions for Oracle as detailed in Section D of the 
Appendix. This backup process utilizes two advanced functions of the ESS's sub system such as 
FlashCopy and Peer-To-Peer-Remote-Copy. 
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4A Oracle Features 

For very large Oracle databases in SAP R/3 production environments, the Split Mirror backup 
capability is essential for the creation of consistent database backup copies without stopping the 
production system. In order to make this possible, Oracle's features such as "ALTER 
TABLESPACE BEGIN BACKUP" and "ALTER TABLESPACE END BACKUP" for 
Online Hot Backup (with Oracle in ARCHIVELOG mode) capability are used to create backup 
copies of the production database without any impact on the production OLTP system, database 
or user activity. During an Online Backup, the Oracle database and the SAP R/3 system remain 
available. All transactions that are logged in the REDO log files during this backup period are 
required to be applied to the backup copy of the database to produce a consistent point-in-time 
copy. 

The ALTER TABLESPACE BEGIN BACKUP command will begin logging entire block images 
on the first change that Oracle encounters on each block owing to the DML activity (such as 
Insert, Update, Delete). This is accomplished by INIT.ORA parameter setting 
_LOGBLOCKS_DURING_BACKUP to TRUE . 

The ALTER TABLESPACE END BACKUP command creates a redo log record containing the 
Oracle marker, BEGIN BACKUP checkpoint, also known as System Change Number (SCN) as 
explained in Section D of the Appendix. This SCN is also recorded in the header of the HOT 
BACKUP data files and ensures that all the redo generated during the backup has been applied to 
the data files. During recovery, as mentioned before, the DBA needs to apply at least the redo 
logs that were generated during the execution of BEGIN BACKUP and END BACKUP 
commands to make the backup data files consistent. It is also necessary to end HOT BACKUP 
mode of the tablespace by issuing the ALTER TABLESPACE END BACKUP command. 

It is strongly recommended that HOT BACKUPS be taken during periods of low DML activity. 
Hence the Split Mirror Backup process, as demonstrated in this paper using the ESS's hardware 
assisted, near-instant local copy (FlashCopy) and remote copy (Peer-To-Peer-Remote-Copy) 
functions, ensures that the backup process can be executed within a very short period of time, 
thus minimizing the impact to the production system. 
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For an overview of the Oracle architecture and other features that take advantage of ESS storage 
management capabilities, please refer to Section A (Oracle Architectural Overview), Section B 
(Oracle Database Growth and Impact on Storage) and Section C (Oracle Memory Management) 
of the Appendix. 

4B FlashCopy - ESS's Advanced Local Copy Functions 

The ESS Specialist identifies the Logical Unit Numbers (LUNs) by their ESS internal serial 
numbers. FlashCopy, ESS's "near-instant" local copy function, can be used for all systems that 
have volumes or LUNs within the same Logical Subsystem (LSS) of an ESS. FlashCopy is set up 
using the Web interface of the StorWatch ESS Specialist. Then, task selections can be made on 
the volume pairs - "FlashCopy" with Full or No Copy, and "WITHDRAW" options. See 
reference [ESS Copy Services] for set up details on FlashCopy. 

The NO COPY option in FlashCopy is useful if the copy operation has to complete within a short 
time so that the source database/application are returned to their normal usage from the end of 
HOT BACKUP mode. The WITHDRAW command in FlashCopy enables the removal of source 
and target volume relationships from a previously specified NOCOPY command. The 
relationship between source and target volumes will automatically end when the physical copy is 
completed. 

Using the ESS Specialist, FlashCopy tasks are created. Then using the Command Line Interface 
(CLI) rsExecuteTask command for the previously defined tasks, the FlashCopy command with 
either No Copy or Physical Copy option is executed. 

FlashCopy, when set up by the StorWatch ESS Specialist, creates an identical copy of the source 
volume on to target volumes when appropriate task is initiated using CLI. Volume identification 
or Disk signatures need to be validated with respect to the host that is connected to the ESS in 
order for that host to start using the target ESS FlashCopy volumes. In order for a single host to 
mount both source and target volumes of FlashCopy pairs, AIX provides the RECREATEVG 
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command, which is packaged as a PTF for AIX 4.3.3 in APAR IY10456. It is officially available 
in: 

1. AIX 4.3.3 Recommended Maintenance Level 05 (RML05) 

2. AIX 4.3.3 RML06 

The RECREATEVG command overcomes the problem of duplicated LVM data structures and 
identifiers caused by a disk copying process such as FlashCppy. It is used to recreate an AIX 
Volume Group (VG) on a set of disks that are copied. Then, the normal commands to bring the 
volume online, i.e. "mounting" can be attempted by the host operating system. 

4C Peer-to-Peer-Remote Copy (PPRC) - ESS f s Advanced Remote Copy Function 

PPRC is a hardware-assisted synchronous remote copy or synchronous mirroring function that 
can help preserve data integrity. Synchronous mirroring means that an I/O is only completed until 
after it is acknowledged from the remote site. 

PPRC is set up on a volume or LUN basis in two or more ESS's, which are connected by ESCON 
(Enterprise System Connection) as shown in Figure 2. Updates to a PPRC volume on the local or 
primary site (primary volume) go first into cache and non-volatile storage (NVS) in the primary 
storage. The updates are then sent over the ESCON links via larger ESCON frame transmission to 
the remote or secondary storage control. When the data is in cache and NVS on the secondary 
site, the receipt of the data is acknowledged and the primary storage control signals to the 
application the completion of the I/O by a Device End status. 
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per 

(2 ports per adapter) 



Figure 2: PPRC Connectivity using ESCON Links 



The enhancements to ESCON protocol, implemented in ESS micro code as an advanced copy 
function, allow an extended distance between two ESS's of up to 103 km, when using multi-mode 
to mono-mode ESCON converters, amplifiers and switches. PPRC can be implemented over 
longer distances using channel extenders from third party suppliers certified on the ESS. 

Up to eight ESCON links are supported between two ESS storage subsystems. The local primary 
storage control with PPRC source volumes and the remote secondary storage control are 
connected via ESCON links. One ESS storage control can act as primary and secondary at the 
same time if it has PPRC source and target volumes. PPRC links are uni-directional, as shown in 
Figure 3, so that a physical ESCON link can be used to transmit data from the primary storage 
control to the secondary. 
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Storage Stfo system 



Storage Subsystem 



Figure 3: PPRC Links between two ESS Storage Sub Systems 



Before PPRC pairs can be established, logical paths must be defined between the logical control 
unit images. The ESS supports up to 16 logical FB control unit images and up to 32 SCSI/Fiber 
controller images. Logical paths are established between control unit images of the same type 
over physical ESCON links that connect Suspending and Resuming Pairs. 

During the suspension of a pair, the primary control unit maintains a bitmap in NVS (ESS's Non 
Volatile Storage located in each of its two symmetric multiprocessing complex that constitute its 
fault tolerant architecture) with a flag bit for each track that was changed on the primary volume. 
This allows for a later ^synchronization (RESYNCH) of the volume pair while allowing only 
cylinders flagged in the bitmap table to be copied to the remote site. 
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As discussed in Section H of the Appendix, the ESS Specialist is a centralized Web-based storage 
management tool providing flexible user access for storage layout, customization, and task 
manipulation required for this SMBR solution. In addition to the Specialist, the ESS functions 
can be administered by the use of CLI at the host level. 

As with FlashCopy tasks, PPRC tasks are created via the ESS Specialist and then invoked from 
the Unix command line using CLI interface. Based on the success of the query - rsExecuteQuery, 
the SMBR automation process can capture the error code for each of the PPRC tasks. 

The features of Oracle and ESS in addition to SAP's requirements to support an automated, 
"lights-out" backup & recovery necessitate planning, set up and understanding of the underlying 
application usage and its performance requirements. 



5 Split Mirror Backup & Recovery (SMBR) Setup 

The SMBR set up involves the physical database design from the SAP installation guide. This 
includes file systems definitions according to sizing (based upon planned usage statistics, I/O 
forecasts, number of users etc.). The file systems requirements follow standard SAP installation 
procedures. 

In a full-featured High Availability SMBR solution we recommend to use two physically 
separated Database hosts and two ESSs, each containing two copies of the production database. 
In our test environment we used two ESS clusters (refer to figures 6 and 7) like separated storage 
systems but without reservations this can simply be extended to Two ESS implementation. The 
installation binaries for SAP kernel and Oracle along with the SSQJ (a load testing tool developed 
by SAP) file systems are also setup for each of the four copies. SSQJ was developed with ABAP4 
in R/3 for benchmarking based on SQL / ABAP statements and table manipulation for 
performance / throughput analysis. 
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The ESS LUNs design is based on logical addressing of striped physical volumes in a RAIDS 
array. The LUN definitions are based on file system requirements and are translated into volume 
groups via IBM's Subsystems Device Driver (SDD) mapping of virtual paths installed on the 
AIX host (see Section F of the Appendix). 

5A SSQJ: R/3 Load Simulation and Testing Tool [12] 

This Split Mirror Backup / Recovery solution utilized the SAP designed SSQJ tool for load 
simulation. SSQJ is a generic test measurement tool that was developed with ABAP/4 in core 
R/3. It enables testing and measurement of the functions in the R/3 Basis system such as runtimes 
of specific functions and their alternatives, resource consumption, query plans in the case of DB 
statements and other functions. SSQJ runs on all SAP-supported database systems with built in 
infrastructure for Measurement & Analysis including History of environment measurements. 

A number of test suites are included in the current version of the package: 

1) SQL statements 

2) ABAP statements 

3) Large Tables for performance and throughput analysis 

4) Archiving Systems 

5) Data Conversion 

6) TPC/D - Benchmark and 

7) Business Warehouse 

SSQJ is currently used for SAP Basis Benchmarking, by SAP Database Porting teams, SAP 
Database Partners and other SAP Hardware Partners. 

The SSQJ tool was used as the kernel database for the SAP/ORACLE SMBR testing. SSQJ 
requires the creation of two new tablespaces, PSAPSSQJD for data and PSAPSSQJI for indexes. 
The size of the SSQJ tablespaces depends on the required database size. For the initial 
installation, we require at least 8GB for PSAPSSQJD and 4GB for PSAPSSQJI. The data files for 
PSAPSSQJD are in /ORACLE/SSD/sapdata3 (total of 8 files of 2GB each), and the data files for 
PSAPSSQJI are in /ORACLE/SSD/sapdata4 (total of 4 files of 2GB each). 
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5B SAP / Oracle File System Definition 

The file systems required for installing SAP system are created on the ESS volume groups. The 
volume groups LSSlOdata, LSS12data, LSSHdata, LSS16data, Sapvg and Logvg as shown in 
Figure 4. First the logical volumes are created using ESS LUNs and then the appropriate options 
are chosen for the logical volumes under AIX using MIN/MAX policies. Finally the file systems 
are created on top of the existing logical volumes. 
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Figure 4: Oracle File Systems for SAP 
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As shown in Figure 4, the volume groups LSSlOdata, LSS12data, LSS14data and LSS16data are 
64GB each. The other volume groups are sapvg and logvg comprising of 1 GB LUNs. sapvg is 
used for the SAP and Oracle executables, and logvg holds the online and archive logs. 

SAP delivers binary executables in order to help achieve online and offline backups as a part of 
the kernel installation. The external data management tools like Tivoli Storage Manager (TSM) 
have products that integrate into SAP R/3 as referred to in Section D of the Appendix. As shown 
in Figure 13, the SAP R/3 objects - Oracle control files, datafiles, archive log files and online 
redo log files are required for a consistent database backup and restore / recovery. 
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Figure 5: Oracle Directory structure in SAP R/3 



Directory and file names are standardized in the R/3 environment as: 
Tablespace files reside in the sapdata<n> directories 

- The online redo log files reside in the origlog and mirrlog directories (mirrored directories) 

- The offline redo log files are written to the saparch directory 

There should be at least three copies of the Oracle control file. 
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The profile init<SID>.ora configures the Oracle instance, and resides in directory 
/Oracle/SSD/dbs: 

- The profile init<SID>.sap configures the backup tools BRBACKUP and BRARCHIVE, and 
resides in directory dbs 

- The profile init<SID>.dba configures the SAPDBA tool, and resides in directory dbs 

- The Oracle alert file is written to directory saptrace / background 

- Trace files of the Oracle shadow processes are written to the directory saptrace / usertrace 
During reorganization, export datasets are written to the directory sapreorg 

The SAP database tools use the directories saparch, sapcheck, sapreorg, and sapbackup. 

As shown in Section D (Topic-SAP R/3 Data layout for Oracle) of the Appendix, consideration 
must be given to sizing recommendations, SAP landscape, and data layout spread evenly across 
storage systems. ESS RAID5 Disk Layout Considerations, as detailed in Section E of the 
Appendix (ESS RaidS Disk Layout Considerations for SAP R/3 Environments), ensure that the 
"hotspot" phenomenon common in non-RAID5 environments are eliminated by striping all 
ORACLE tablespaces across all arrays, thus utilizing the cumulative throughput of all device 
adapters. 

At this point we complete the installation of Oracle 8.0.6,3 and SAP Version 4.6B for ADC host 
joyjoy. The SAP / Oracle System / Instance name is "SSD" and the system number NR=00. 

5C Split Mirror Infrastructure Setup 

The SMBR live database, as shown in Figure 6, consists of DB A for the SAP Instance "SSD " on 
the primary ESS attached to host joyjoy. it contains all the Oracle binaries, SAP kernel, transport 
directory, Oracle Online / Archive log directories, Oracle instance directory, Oracle file systems 
and SSQJ file systems. 
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DB C is the primary PPRC mirror of the live database DB A . DB B is the Safety FlashCopy copy of 
the live database DB A . This safety copy is created in order to maintain a Point-in-time-copy of the 
live Production Database DB A , should there be any problems encountered with DB A during the 
SMBR operation. 




Figure 6: R/3 Split Mirror Backup & Recovery and Backup Host 



In situations where the customers' BASIS or DBA group would like to mount the file systems for 
verification purposes by using the FlashCopy or PPRC target volumes on the same host, the 
Physical Volume Identifier (PVID)'s need to be renamed using the AIX RECREATEVG 
command as mentioned earlier in Section 4B. 
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5D ESS LUN Definition 

For all references to LUN and volume group sizes, 1GB translates to 1*1000*1000*1000 bytes. 
All the LUNS for host "joyjoy" are defined only on cluster 1 for our SMBR solution scenarios 
using FlashCopy and PPRC. Each of the 8 ranks on Cluster 1 consists of twelve 8GB LUNS. In 
addition to these, two 1GB LUNs are also defined on each of the ranks. The 8GB LUNS are used 
for the Oracle data files, while one set of eight 1GB LUN is used for online redolog and archive 
saparch files. The other set of eight 1GB LUN is used for the SAP R/3 and Oracle binaries and 
other temporary storage. 

OSPL2 and OSPL6 are the production and disaster recovery ESSs in our setup. 

For the initial installation, four 8GB LUNs from each rank are used to layout the SAP R/3 Oracle 
data files. This leaves the other eight - 8GB LUNs for FlashCopy (safety copy) and PPRC. 

As shown in Figure 7, there are a total of four LSSs, 4 on each cluster of an ESS. Each LSS on a 
cluster has two ranks assigned to it. There are two sets of four 8GB LUNs of an LSS that are 
assigned to one volume group. In our case, that translates to four volume groups of 64GB each, 
one for each LSS. The volume groups are LSSlOdata, LSS12data, LSS14data and LSS16data on 
ESS1 (OSPL2). Similarly, LSS1 Idata, LSSHdata, LSSl.Sdata and LSS17data volume groups are 
created on ESS2 (OSPL6). 

The other volume groups are sapvg and logvg comprising of 1GB LUNs. sapvg is used for the 
SAP and Oracle binaries, and logvg holds the online and archive logs. 
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Loop Rank 



Disk Adapter 

Figure 7: ESS Logical Sub Systems (LSS) 

In an optimal database layout where the distribution of tablespaces follows the basic principle - 
spread the data over as many physical disk / arrays as possible [4, 6, 7] - Tablespaces. should be 
extended by at least one full stripe set across all arrays. 

This ensures that tablespaces remain distributed across a stripe set. Allocation of a specific 
number of arrays for a tablespace facilitates distribution and placement of the data files. The more 
arrays that are allocated, the better the distribution is. Data file distribution should be based on a 
round-robin placement across ESS clusters and arrays and the granularity can be achieved using 
AIX Logical Volume allocation. 
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In the SMBR Solution validation scenarios, the AIX Logical Volume Manager maps the assigned 
ESS 8GB LUNs as hdisks. Grouping the LUNs of a single RAID array can create volume groups. 
Under this traditional route for database layouts, AIX file systems are created over AIX logical 
volumes that reside on a single array. The data files of a tablespace are then distributed over file 
systems on different arrays. The criterion for a data file to be placed in these file systems is still 
the same - the overall I/O activity should be distributed across all available arrays. Creation of 
volume groups and logical volumes should be within the constraints imposed by the AIX Logical 
Storage Management. 

5E Volume Group Assignment 

Once the LUNs are defined on each ESS and assigned to hosts - joyjoy and decme, the LUNs are 
made available to joyjoy by running the 'cfgmgr' command on that host. The Subsystems Device 
Driver (SDD) - is a high availability automatic I/O Path Failover Function that provides 
management of active paths to the LUNs as outlined in Section F of the Appendix. The SDD 
software needs to be installed on the hosts before running cfgmgr. 

The LUNs appear as hdisks on joyjoy. If a LUN is assigned to one path, an hdisk, let's say hdisk6 
is defined on joyjoy. As joyjoy has four SCSI adapters, the LUNs are configured such that they 
can be accessed through all four paths. For each additional path, another hdisk is assigned. So for 
four paths, we have four hdisks all pointing to the same LUN on the ESS. 

When SDD is used, additional data path devices called vpath devices are created. Each hdisk set 
(a set is based on the number of paths from the host to the LUN) is assigned a vpath device. In 
our example, vpathl will consist of hdisk6, hdisk46, hdisk90 and hdiskl26. 

When a volume group is to be created on joyjoy, two options are available in AIX administration 
utility smitty: 

1) Add a Volume Group, or 

2) Add a Volume Group with Data Path Devices. 

As we are using SDD, the volume groups are created with the second option. 
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Volume Layout for Oracle 

The volume layout for SAP R/3 system in Oracle for ESS is depicted in Figure 17 (Logical Sub 
System vpath mapping to hdisks) of Section H of the Appendix. 

6 Split Mirror Backup & Recovery Process 
6A Start Situation of Split Mirror Backup Process 

The live system is in normal READ / WRITE operation state. ONLY the Log Volumes (Online 
Redo Log Files) are in a constant synchronous PPRC connection between source DB A and target 
DB C across ESS1 and ESS2 as shown in Figure 8. 

The constant synchronization of log volumes between ESS1 and ESS2 will ensure that the 
disaster recovery site is up-to-date with the transactional changes to the database, making any 
recovery from an apparent failure on the primary database simpler. Furthermore, possible data 
inconsistencies due to user or program errors are not immediately copied to the remote data files, 
but with the constantly synchronized Logs we are able to recover to any point in time. 

The periodic PPRC re-synchronization of the data volumes during the SMBR activity will ensure 
that any structural changes (e.g. adding datafiles to tablespaces etc.) including all de-staged data 
are mirrored on the remote site. 
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Figure 8: Split Mirror Backup & Recovery and Standby R/3 System 



The SMBR process steps (nine) depicted in Figure 8 are listed below: 



1: FlashCopy (FC) DB A to DB B (Safety Copy) 

2: Resync DB A to DB C 

3: Withdraw Prior FC DB D 

4: Alter Tablespace Begin Backup on DB A; Freeze PPRC pairs 

5: Create Data Volume FC DB D ; Alter Tablespace End Backup; Switch Log File 

6. Resync PPRC Log Volumes and Create Final FC 

7: Backup DB D to EBT (Enterprise Backup Tool) / TSM 

8: Mount FC Volumes to Backup Host "decme" 

9: Start R/3 Oracle on Backup Host 



After the prior SMBR backup activity, the DB D instance is in FlashCopy No Copy relationship 
with DB C while DB B is in FlashCopy No Copy relationship with DB A . 
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DBd volumes have been used to bring up a SAP instance after successful backup to tape. Using 
the SAP R/3 Homogeneous System Copy procedures for post copy BASIS administration tasks 
such as changing the RFC settings, locking user id's, TMS setup and batch job cancellations etc., 
will ensure that the actual production related activities are deactivated in the test SAP instance on 
DB D volumes. 

The first step in SMBR process would be to logoff all the users from the Production-Fix / 
reporting / standby instance i.e. DB D (on the backup host decme) before stopping respective SAP 
and Oracle processes. The SAP / Oracle instance on the Safety FlashCopy target volumes DB B in 
the primary ESS is also stopped. 

The next step is to collect all the current layout details from within the ORACLE database 
through the dynamic SQL statement (PL/SQL) that can query the latest information on the 
database structure including the log group, tablespaces, and data file information. Dynamic SQL 
statements are embedded in the source SQL statement and are stored as character strings input to 
or built by the source program at run time. 

During this step, the file structure information and volume group layout along with device list 
from the host operating system is obtained. This step will confirm that the current users in the 
SAP instance SSD on the decme host (probably the Production-Fix system users) are logged off 
and the Journaled File Systems (JFS) on decme will be unmounted. All the application related file 
systems, except those that pertain to the AIX host operating system on the decme host, are also 
unmounted. 

General comments on the Backup Process: 

1 Begin Phase: 

- Check for status of FlashCopy relationships between DB A & DB B , DB C & DB D 

Verify if database on DB C was opened and if data volumes between DB A and DB C are out of 
sync (to ensure minimization of error propagation) 

- Check that the log volumes between DB A and DB C are in synchronous PPRC mode 
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2 FlashCopy Phase in ESS: 

- Withdraw prior FlashCopy relationships between DB A & DB B 

- For all the tablespaces issue HOT BACKUP commands 

- FlashCopy DB A to DB B 'Data volumes and Oracle & SAP binaries' in no-copy mode (copy 
operation of data files, redo log / saparch takes place at different instants in time) 

- End HOT BACKUP mode for all tablespaces. Issue log-file switch and control file copy 
commands 

- FlashCopy DB A to DB B 'Log Volume group' in no-copy mode 

3 PPRC and FlashCopy Phase between ESS1 / ESS2: 

- If the suspension of PPRC between data volumes and log / saparch / sapreorg volumes occurs 
at different instants in time during this step, there is a requirement to execute switch log file / 
backup control file commands after the end of HOT BACKUP mode 

- However, with the PPRC subsystem level 'freeze' command / operation, which is an atomic 
suspension of all PPRC disk-pairs, the requirement to do a recovery using backup control file 
/ redo log files is eliminated 

- FlashCopy DB C to DB D 'Data volumes and Oracle & SAP binaries' in no-copy mode (copy 
operation of data files, redo log / saparch takes place at different instants in time) 

- End HOT BACKUP mode for all tablespaces. Issue log-file switch and control file copy 
commands 

- At the end of hot backup process issue log switch / backup control file commands 
Resynchronize PPRC 

- FlashCopy DB C to DB D 'Log Volume group' (Archive logs) in no-copy mode 

4 Copy DB D to Tape Phase using a backup utility: 

- Mount DB D volumes to host decme. Irrespective of FlashCopy option used (physical 
complete or no-copy mode), all the volume sets on DB D are backed up to tape using a backup 
utility 
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5 Second Instance Phase: 

Start SAP instance on DB D using 'SAPDBA -startup*. This verifies that SCN (System Change 
Number) is consistent with Datafiles and Control files. It also resets the data files to end 
backup mode and opens the database after rolling back any un-committed transactions 
between Begin Backup and End Backup modes 
- Starting Oracle through 'SAPDBA -startup' command will verify that DB D is a consistent 
copy 

Based on customer requirements, the second instance can be provided with a change in SID 
name using the output from Oracle's 'BACKUP CONTROL FILE TO TRACE' command 
output 

6 New Backup cycle Initialization Phase: 

In order to be prepared for a new backup cycle, the systems (DB B , DB C and DB D ) must be set 
back to the start situation 

In this SMBR set up, the following tasks were created using ESS Specialist / CLI 
command sets: 

FCAllNoBgCpWESSl: Withdraw previously held FlashCopy NoCopy relationship between 
FlashCopy source/target pairs DB A and DB B in ESS1 . 

FCAllNoBgCpWESS2: Withdraw previously held FlashCopy NoCopy relationship between 
FlashCopy source/target pairs DB c and DB D in ESS2. 

FC1016NoBgCpESSl: FlashCopy source production Data volumes to target Safety copy Data 
volumes in LSS10, LSS12, LSS14, and LSS16 in ESS1 

FC1117NoBgCpESS2: FlashCopy source production Data volumes to target Safety copy Data 
volumes in LSS1 1 , LSS13, LSS15, and LSS17 in ESS2 
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FCLogVNoBgCpESSl: FlashCopy source production Log volumes in DB A to target Safety copy 
Log volumes (saparch, sapbackup, sapreorg, log directories) in DB B in ESS1 

FCLogVNoBgCpESS2: FlashCopy source production Log volumes in DB c to target Safety copy 
Log volumes (saparch, sapbackup, sapreorg, log directories) in DB D in ESS2 

EstALL4PATH: Establish PPRC paths and/or ensure that the paths exist between ESS1 (LSS10, 
LSS12, LSS14, LSS16) and ESS2 (LSS1 1, LSS13, LSS15, LSS17) 

ResyncPPRCAll: Resynchronize PPRC source and target volumes in between ESS1 & ESS2 

ResyncPPRCLogVs: Resynchronize PPRC source and target Log volumes in between ESS1 & 
ESS2 

Frzall: Subsystem level withdrawal of paths and dataflow between PPRC source and target 
volumes 

OSPL2C0 is the Cluster 0 in ESS1 
OSPL6C0 is the Cluster 0 in ESS2 

rsExecuteTask.sh is a CLI command to be initiated from AIX host 
CLI command option -v provides verbose output 
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6B SMBR Implementation Steps 

Each SMBR step of the backup process is described as follows: 
Step 1: Create a Safety FlashCopy of the Production Instance 

Create Safety copy of the production database DBa 

i. Withdraw previous Safety Copy source / target FlashCopy relationships on the Primary 
ESS1: 

To Withdraw previous Safety Copy source / target FlashCopy relationships on the 
Primary ESS1, as captured in a Task "FCAUNoBgCpWESSl", the following CLI is 
initiated against OSPL2C0 from AIX: 

rsExecuteTask.sh -v -s ospl2c0 FCAUNoBgCpWESSl 

OSPL2C0 is the cluster 1 of ESS 1 that hosts the Production Volumes. 

FCAUNoBgCpWESSl is the predefined task set up between source Production volumes 
and Target Safety FlashCopy volumes. 

ii. Reads and writes to the Production SAP / Oracle database DB A continue as usual on all 
the tablespaces while Oracle is put in HOT BACKUP mode using the following Oracle 
command: 

ALTER TABLESPACE <PSAPBTABD> BEGIN BACKUP; 

iii. Flash Copy all DB A data volumes to DB B volumes with No Copy option: 
rsExecuteTask.sh -v -s ospl2c0 FC1016NoBgCpESSl 



TECHNOLOGY 



Page 31 



Storage Management for SAP and Oracle: Split Mirror Backup / Recovery with 
IBM's Enterprise Storage Server (ESS) 

v. Upon successful completion of FlashCopy, bring the database back to normal mode 
using Oracle's END BACKUP command. This ends the Online HOT BACKUP mode 
for Oracle after which checkpoints are completed and a "redo log switch" is initiated for 
each of the redo log files. The following Oracle commands are executed: 

ALTER TABLESPACE <PSAPBTABD> END BACKUP; 

ALTER SYSTEM SWITCH LOGFILE; 

ALTER DATABASE BACKUP CONTROL FILE 

TO /oracle/SSD/sapreorg/bakCntrlSSD.dbf; 
ALTER DATABASE BACKUP CONTROL FILE TO TRACE; 

The output of the ALTER DATABASE BACKUP CONTROL FILE TO TRACE 
command creates a trace file that can be used to change SID name of the DB B volumes. 

v. After FlashCopy of the Data volumes and the end of Oracle's HOT BACKUP mode, 
execute FlashCopy of the log volume group that consists of archive log files, sapbackup 
& sapreorg directories and redolog files: 

rsExecuteTask.sh -v -s ospl2c0 FCLogVNoBgCpESSl 

DB B now contains the latest Archived log files from DB A . 

This step completes the tasks required to create a safety copy on the source ESS1 storage 
subsystem. This safety copy will enable us to recover the database with ORACLE 
recovery command executed on the DB B volumes, should DB A volumes become 
unavailable during this process. 

Extract all the system information pertaining to disk storage from within AIX on joyjoy 
and remote copy those files to host decme. This will help bring up an SAP instance on the 
DB B volumes. In case of corruption to DB A or DB C volumes during the SMBR activity, 
the DB B volumes can be used as a safety measure. 
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Step 2: Resynchronize DB A to DBc 

i. Ensure that the ESCON Paths exist (and are active) between PPRC source (DB A ) and 
target (DBc) volumes; 'establish' if they do not already exist: 

rsExecuteTasLsh -v -s ospl2c0 EstALL4PATH 

ii. Resynchronize all the PPRC source (DB A ) and target (DB C ) volumes (this includes log and 
data volumes): 

rsExecuteTask.sh -v ~s ospl2c0 ResyncPPRCAll 

iii. Monitor percentage of Resync completion 

Step 3: Withdraw previously held FlashCopy (source / target) Volumes on ESS2 

after stopping the R/3 Application and Oracle Database on the Backup host 

In ESS2, withdraw previously held FlashCopy No Copy relationship between DB C and 
DB D volumes from the previous backup run: 

rsExecuteTask.sh -v -s ospl6c0 FCAllNoBgCpWESS2 

(In the previous backup run, we made a safety copy DB B in ESS1 and a final flash copy 
DB D in ESS2.) 

Step 4: Confirm that the Production Database tablespaces are in normal mode; 
Initiate HOT BACKUP mode; Suspend all the PPRC source (DB A ) and 
target (DBc) volume pairs. 

Put Oracle in HOT BACKUP mode for all the tablespaces (DB A ) using the following 
Oracle command: 

ALTER TABLESPACE <PSAPBTABD> BEGIN BACKUP; 
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This will permit all write activity on DB A to be written directly to datafiles so that DB C 
(PPRC target volumes) can function as a consistent FlashCopy source for the final Split 
Mirror copy to be created on DBd. 

In order to create a consistent database copy (FlashCopy) DB D , the PPRC relationship 
between DB A and its mirror DB C has to be frozen. 

The entire subsystem level freeze activity of PPRC pairs between production - DB A and 
mirror - DB C will ensure that the data and log volumes can be suspended in one atomic 
operation, thus ensuring a consistent database copy. This atomic split capability of PPRC 
provides critical time saving functionality for high availability operations so that the 
Disaster Recovery site has the latest synchronous PPRC copy, enabling rapid restoration of 
this synchronous mirror for production purposes. The execution of the "Freeze" command 
at the subsystem level instead of the "Suspend" (another PPRC capability) command at 
LUN level speeds up the operations because it avoids issuance of multiple commands to 
individual LUN relationships between PPRC source and target volumes. 

Thus the Freeze capability of the ESS subsystem enables the rapid creation of a consistent 
data base copy during the SMBR process. 

(a) Freeze all the ESS subsystem level PPRC relationships between ESS1 and ESS2. This 
activity stops all data flow between paired data and log volumes: 

rsExecuteTask.sh -v -s ospl2c0 FrzAll 

(b) Verify the status of the task completion using rsExecuteQuery in a query loop. Obtain 
the error code and send an automated message upon successful task completion. 
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NOTE : Maintaining log volumes in continuous synchronous state between ESS1 and ESS2 

If constant remote mirroring is an absolute customer requirement for mission critical 
environments, we recommend that the log volumes be kept in a constant PPRC state. ESS is able 
to perform a FlashCopy from a PPRC target to another volume without PPRC suspension. In this 
case, if we still want to use the Freeze command for data volumes, both data and log volume 
groups must be stored in separate LSSs (as shown in Figure 4). 

In this SAP/Oracle implementation, because of the time lapse between the moment of PPRC 
freeze of all data and log volume pairs at the sub system level, and the completion of the logical 
FlashCopy (NoCopy) of the data volumes from DB C onto DB D > the remote vaulting of log 
volumes is momentarily suspended during this process. However, as mentioned above, if remote 
log mirroring via PPRC during a selected SMBR window is an absolute customer requirement, 
then use of the PPRC SUSPEND command (at the LUN level) only for the data volume pairs, as 
described in Step 4, is recommended. 



Step 5: Create Data Volume FlashCopy DB D ; Alter Tablespace End Backup; 
Switch Log File 

Proceed with the FlashCopy of data volumes. Initiate FlashCopy of the source DB C to 
target DB D in NoCopy mode: 

rsExecuteTask.sh -v-s ospl6c0 FC1 1 17NoBgCpESS2 

Verify error codes for the FlashCopy. Upon error code of zero, bring the database back to 
normal mode using Oracle's END BACKUP command. This ends Online HOT BACKUP 
mode of DB A after checkpoints are completed and a "redo log switch" is initiated for each 
of the redo log files. The following Oracle commands are executed: 

ALTER TABLESPACE <PASAPBTABD> END BACKUP; 
ALTER SYSTEM SWITCH LOGFILE; 
ALTER DATABASE BACKUP CONTROL FILE 

TO/oracle/SSD/sapreorg/FcCntrlSSD.dbf; 

nCHfJDIOGY 



Page 35 




Storage Management for SAP and Oracle: Split Mirror Backup / Recovery with 
IBM's Enterprise Storage Server (ESS) 

ALTER DATABASE BACKUP CONTROL FILE TO TRACE; 

The output of the ALTER DATABASE BACKUP CONTROL FILE TO TRACE 
command creates a trace file that can be used to change SID name of the DB D volumes, 
thus facilitating the instantiation of a secondary instance. 

Step 6: Resynchronize PPRC source and target log volumes and 
Create Final FlashCopy 

After successful completion of FlashCopy (logical copy) and Oracle's END BACKUP 
command on the Production instance, it is time to reinitiate the PPRC links between ESS1 
and ESS2 (which were in Freeze mode in Step 4). 

i. Initiate PPRC links and establish paths between OSPL2/ESS 1 and OSPL6/ESS2: 
rsExecuteTask.sh -v -s ospl2c0 EstA114Path 

ii. Resynchronize all the incremental redolog activity in ESS1 (as in the execution of Step 5 
above) during the period of PPRC freeze: 

rsExecuteTask.sh -v -s ospl2c0 ResyncPPRCLogVs 

Keeping the Log & Data volumes in sync at different times and only synchronizing data 
volumes during the periodic SMBR activity will ensure that the Disaster recovery site 
(ESS2) has log volumes that can be either rolled forward or rolled backward as per the 
business requirement, in the event of a disaster. 

After FlashCopy of the Data volumes (as in Step 5 above) and at the end of Oracle's HOT 
BACKUP, execute FlashCopy of the log volume group that consist of latest archive log 
files, sapbackup & sapreorg directories and redo log files: 
rsExecuteTask.sh -v -s ospl6c0 FCLogVNoBgCpESS2 
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The FlashCopy of Data volumes completed in Step 5, followed by the FlashCopy of Log 
Volumes completed during this step, creates the Final Point-in-Time Split Mirror of 
Production Database. 

As mentioned in Section D of the Appendix on the SPLIT BLOCK phenomenon, data file 
changes, because of update / delete activity in Oracle, will prompt for redolog changes that 
in turn are not captured in the control file for the SCN changes. As the PPRC subsystem 
level Freeze is an atomic operation, the control file on ESS1 is merely backed up and is not 
necessary for any type of recovery. The datafiles captured at the moment of PPRC freeze 
and log volume group captured after the logswitch/control file copy will provide a 
consistent database copy. However, during this finite period of SMBR activity, if there are 
any structural changes (such as adding datafiles, adding tablespaces etc.) made to the 
database, it is necessary to resynchronize the data volumes as in Step 2. For a successful 
SMBR completion, ensure that there are no structural changes made to the database during 
this period. 

This step completes the tasks required to create a Split Mirror copy of the Production 
environment with Oracle's Online backup functionality. 

Upon the completion of this step, all SAP production operations landscape is in normal 
state as at the start situation described in section 6A. 

Optional Physical Flash Copy of DB D Volumes: 

If there is a requirement for a complete physical copy of the final FlashCopy (logical copy) 
DB D volumes, initiate a FlashCopy with Copy option from DB D to DB E (not shown in the 
Figure 8) in ESS2 as an extension to Step 6. 

i. Initiate Physical FlashCopy of DB D to DB E in Copy mode. 

Now continue with the Steps 7, 8 and 9. 
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Remote copy all the file systems, volume group, and device lists from the source 
production host 0°yj°y)- 

Step 7: Backup the Final FlashCopy volumes to a backup utility 

At this point using SAPDBA / BRBACKUP / TDP for R/3, Legato or any compatible 
backup utility program, backup the DB D instance's SAP / ORACLE objects to tape. We 
' will always save all the DB D volumes to tape media by invoking complete file system 
backup or a backup utility. 

Step 8: Mount FlashCopy volumes (on ESS2 / OSPL6) 

Rediscover all the devices that are part of the defined volume groups at the host operating 
system level. Varyon all the volume groups and mount the file systems required for 
ORACLE & SAP on host decme. 

Varying-on and mounting of all the relevant volume groups is done after rediscovering 
the physical volumes and remapping those to the virtual paths as seen by the Host 
operating system via the SDD. 

Step 9: Start SAP / Oracle on the Backup Host for verification purposes 

In order for us to verify that it is a consistent copy of the live database, this step is optional. 

Startup Oracle in nomount mode and bring all the datafiles from BEGIN BACKUP mode 
to normal mode. 

ALTER DATABASE DATAFILE 7oracle/SSD/sapdatal/btabd_l/btabd.datal' END 
BACKUP; 

Startup Oracle using "sapdba -startup" or by server manager. 
Startup SAP after checking for Oracle error logs. 



TE&HN010&T 



Page 38 



Storage Management for SAP and Oracle: Split Mirror Backup / Recovery with 
IBM's Enterprise Storage Server (ESS) 

Changes to the SMBR Solution for Oracle 8.L6 and later releases: 

Subsequent to our SMBR solutions validation on Oracle 8.0.6, the latest features from Oracle for 
the 8.1.6 and later releases provide the following two important commands: 

a) ALTER SYSTEM SUSPEND: Suspend or stop all I/O to data and control files and restrict 
access to the database. All ongoing I/O operations are allowed to complete but all new / incoming 
database accesses are placed in a queued state. This command should be preceded by the 
ALTER TABLESPACE <PASAPBTABD> BEGIN BACKPUP command. 

b) ALTER SYSTEM RESUME: This command makes the database available for resumption of 
queries and I/O activity. This command stiould be followed by ALTER TABLESAPCE 
<PASAPBTABD> END BACKUP. 

With the availability of these two key functions, the suspension / freeze of the PPRC pairs 
between ESS1 and ESS 2 will no longer be necessary in our SMBR process. 

Accordingly the following changes will be required to the steps in Section 6 (Split Mirror 
Backup & Recovery Process) of our SMBR process: 

In Step 1 item ii, execute command ALTER SYSTEM SUSPEND after the ALTER 
TABLESPACE <P A S APBT ABD> BEGIN BACKUP command. 

In Step 1 item iv, execute ALTER SYSTEM RESUME before ALTER TABLESPACE 
<PASAPBTABD> END BACKUP command. 

In Step 4, because of this new SUSPEND feature of Oracle 8.1.6, the PPRC Suspension or 
Freeze steps are NOT necessary. 

Execute ALTER SYSTEM SUSPEND after ALTER TABLESPACE <PASAPBTABD> 
BEGIN BACKUP command. 
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In Step 5, execute ALTER SYSTEM RESUME before ALTER TABLESPACE 
<PASAPBTABD> END BACKUP. 

Execution of Step 6 is not necessary. 

The updated version of SMBR for Oracle on AIX with the ESS will incorporate Oracle's 
new SUSPEND / RESUME features in detail. 

7 Recovery 

The backup process described above will provide a consistent copy of the live database. This 
backup file set will be on tape media and on the DB D volumes of ESS2 (OSPL6). 

7A Recovery Considerations for Split Mirror Backup Process 

The database recovery is performed in several steps: 

i. For a recovery process, the tape backup needs to be first restored on to the DB D volumes 
on ESS2 / OSPL6 and then the DB D volumes can be FlashCopied on to the PPRC 
volumes (DB C ) on ESS2 / OSPL6. The recovery steps further involve executing PPRC 
from source ESS2 / OSPL6 to the target volumes on ESS1 / OSPL2 if DB A rebuild is 
required. 

ii. Depending on the Service Level Agreements (SLA) consistent with Customer 
Landscapes and Requirements, we can repeat the SMBR steps in reverse order of 
execution (Steps 1 through 9 above). This will ensure recovery using a consistent 
database copy from tape media or from DB D volumes on ESS2 / OSPL6. It will also 
provide a completely recovered SAP SSD instance on ESS1 / OSPL2. 

iii. During the roll forward process, all committed transactions are reapplied to the database. 
At the end of the roll forward, a list of open transactions exists. These transactions are 
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still awaiting a commit. For data consistency reasons, open transactions will now be rolled 
back, to make sure that any un-committed changes are not applied to the database. 

While building the recovery scenarios, based upon customer requirements, ESS Specialist- 
based tasks should be created to execute the recovery processes. 

7B General Considerations for SAP / Oracle Recovery Scenarios 

The backup process described in Section 6 above will provide a fuzzy copy of the live database, 
which will be required to be made consistent by the application of the archive logs generated 
during the period of Oracle's HOT BACKUP state. This backup file set will be on tape media and 
on the DB D volumes of ESS2 (OSPL6). 

An AUTORECOVERY process for SAP R/3 using ORACLE brings the database to the latest 
consistent state based on the disk image and the online redo log files. This recovery uses entries 
in the control file and the online redo log files. If a disk error occurs (in a non-RAID5 sub system) 
and the contents of the disk are not recoverable, AUTORECOVERY will not work. This applies 
to the data files and online redo log files of the database. Therefore, as discussed in Section e of 
the Appendix, "ESS Raid5 Disk Layout Considerations for SAP R/3 Environments", using ESS's 
RAIDS striping, provides a highly available disk subsystem to protect log files and data files. 

The contents of DB D and DB C volumes of the SMBR process differ from each other in that the 
restart of DB D SAP instance (SSD) will roll back all the transactions that were open at the time of 
HOT BACKUP mode. Also, because of the RESYNC state in Step 6 of the SMBR process (in 
Section 6), DB C and DB A volumes are a pair of synchronous log mirrors and differ from the DB D 
(PoinMn-Time copy) volumes. 

The DB D volumes can be used for multiple purposes such as: 

- Database Validity Checks 
Database Reorganization dry-run tasks 

- Testing SAP support packages 

- Training 

- Applying of OSS Notes 
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Production-Fix system 
Reporting Instance 

The DB D FlashCopy volumes can be also set up to serve as a Standby Database by: 

a) mounting the volumes to host decme 

b) placing the DB D in roll-forward pending mode and then roll-forwarding the mirror. 

In order to create a second instance to be accessed by SAP R/3 Test Instance the following is 
suggested: 

- Generate a control file script by issuing ALTER DATABASE BACKUP CONTROLFILE 
TO TRACE right after ending hot backup mode on production database on OSPL2. 

Let the second host decme gain access to the FlashCopy image DB D in ESS2 by mounting it. 
Make the FlashCopy image area write-able by the second host decme. 

- If the second host decme accesses the datafiles in the FlashCopy area with a path different 
from the primary host, edit the control trace file so that it points to the right data files. 

- Create control file with resetlogs option, and recover the database by applying the archived 
redo logs so that the database has cleared the HOT BACKUP fuzziness. 

Open database with resetlogs option. 

8 Process Automation - Solution Integration in Customer 
Environments 

In establishing this SMBR solution for R/3 with SSD and ESS, IBM has created end-to-end 
automated procedures that will enable this solution to seamlessly integrate into customer 
environments. While recognizing that each customer will likely implement this solution uniquely, 
the procedures incorporate use of platform-independent Java technology, which will execute the 
customized set of Split Mirror routines for a particular backup environment. Inherent benefits of 
Java technology implementation will accrue to the customer in terms of flexibility, security, and 
location transparency. With this standards-driven approach, customers can integrate this solution 
with customer-preferred enterprise solutions management consoles (TME10, Open View, BMC 
etc). 
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Appendix 
A) Oracle Architectural Overview 

An Oracle database system has three processes that write information from the Shared Global 
Area (SGA) to the appropriate files: 

- To accelerate the writing of checkpoints, the Checkpoint Process (CKPT) is started. 
During a checkpoint, the database writer (DBWR) asynchronously writes the changed data 
blocks from the SGA to the database data files. 

- The logwriter (LGWR) synchronously writes the changed log records from the SGA 
redo log buffer to the currently active online redo log file. 

In a production database environment, the database system must always run in ARCHIVELOG 
mode. An archiver (ARGH) process archives a completed online redo log file into the offline redo 
log file in the archive directory - /oracle/SSD/saparch 

In addition, when an Oracle database instance is started, several other processes are created: 
SMON, PMON, RECO, and LCK. 
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Figure 9: Oracle Overview for SAP R/3 



Legend: 



The Oracle database is stored in 4 / 8 / 16KB blocks in data files on disk. In order to accelerate 
read / write access to the data, these data blocks are cached in the database buffer pool in 
production CPU main memory. The Oracle database management system holds the executable 
SQL Statements in the Shared SQL Area, which is part of the shared pool. 
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SAP/Oracle Work process management: 

Each SAP R/3 Dialog, Batch, Update, and Spool work processes in an SAP instance use TCP/IP 
and TNS listener to: 

- Connect to the database as one SAP R/3 user 

- Handle database requests for the different R/3 system users 

- Communicate with a corresponding shadow process on the database 

Oracle's Dedicated Shadow Processes are created when a new user establishes a session with the 
R/3 system. The Shared Processes are required by the DBMS systems to function arid perform 
various database management tasks. 

The various Oracle processes that impact disks are depicted in Figure 10: 
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Data files Log Files Log Files 



Figure 10: Overview of Oracle 'write* process to Disks 

B) Oracle Database Growth and Impact on Storage 

The Oracle database uses tablespaces, which hold the database objects, such as tables and 
indexes. On disk, a tablespace consists of one or more data files. Adding datafiles to it can 
increase the capacity of a tablespace. The R/3 naming convention for tablespace names is 
PSAP<tablespace_name><extension>. The abbreviations in the tablespace name are part of the 
data file name. 
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Depending on the size of each data record, several data records can be stored in an Oracle data 
buffer. Within a tablespace, the extents belonging to different data objects compete for storage 
space. SAP R/3 for Oracle on non-RAID5 storage systems separates data and index objects into 
different tablespaces. 

In non-RAID5 storage architecture implementations, the emphasis is to distribute data and index 
tablespaces to different physical devices, thus improving database performance. With ESS's 
robust RAID5 architecture and its use of striping different types of data on all RAID arrays, the 
performance hot spots are eliminated while I/O performance improves through parallel access to 
all members of the array. 

Oracle stores tables and indexes in individual data blocks. When new storage space is required for 
a table or an index, one or more contiguous data blocks of a data file are allocated to form an 
extent. Each table and index is assigned to a tablespace, which consists of one or more data files 
at the operating system level. All table and index data is stored in the data files of the tablespace. 

Several storage parameters influence the growth of Oracle data objects. During installation of an 
R/3 system, while creating an SAP table or index, the default storage parameters - INITIAL, 
NEXT and MAXEXTENT values are used. The first extent (INITIAL EXTENT) should be large 
enough for the expected table or index size. If an extent of a data object becomes full during an 
insert or update operation, the Oracle storage management system attempts to allocate another 
extent in the tablespace. 

The growth of an object, such as a table, index, or rollback segment, is determined according to 
the size specified by the NEXT parameter. 

Oracle allocates extents to an object up to the limit of the MAXEXTENTS parameter. When the 
maximum number of extents for an object is reached, the MAXEXTENTS parameter can be 
increased after checking the . size of the table's NEXT parameter and setting the AUTOEXTEND 
parameter to true. Under special circumstances, depending on the nature of the tables (static or 
dynamic) three additional storage parameters PCTFREE, PCTUSED, and PCTINCREASE are 
applied. 
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Tablespace Reorganization addresses the problem of fragmentation to optimize the amount of 
storage space. Extents are merged together to reduce the number of extents in the database and 
some data files are merged together to reduce the number of data files in the database. 

Online Reorganization should be done during light transaction load on the tables to be 
reorganized. However, very large tables should be reorganized using export / import 
functionalities of Oracle. Block usage for the reorganized object will be optimal after a table or 
index is reorganized. Additional storage space is required to perform a re-organization and to 
store intermediate data, in database or in /oracle/SSD/sapreorg directory. 

Performance Considerations in Database Layout 

Customer experiences have shown that poor database performances at SAP installations often 
result from high I/O wait times for database access. I/O contention in a non-RAID5 environment 
can be traced to: 

Inefficient application design (expensive, unnecessary, and poorly qualified statements). 

Data not evenly distributed across many disk cylinders. 

Heavily accessed tables or indexes not distributed or striped across many disks. 

Oracle database and file objects (tablespaces and directories) not optimally laid out, thus 

causing numerous Oracle threads or processes to access the same disks as shown in 

Figure 1 1 . 

In the past, the database layouts in non-RAID5 environments, have often tried to separate 
tablespaces onto different devices, e.g., separating data and index tablespaces and putting each 
data tablespace on different drives. The consequence was that the database often ended in 
hotspots on devices that carried a single tablespace. Manual intervention was necessary in each 
case to alleviate the problem. Additional performance degradations occur over time due to 
changing application profiles and data growth leading to more hotspots and restructuring 
demands. 
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Figure 11: SAP R/3 and Oracle Read/Write Access - Impact on Disk Arrays 



With advances in storage subsystem technologies and in particular advanced RAIDS 
implementations such as in the ESS, applications can now experience significant performance 
improvements, high availability and recoverability. 
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C) Oracle Memory Management 

As shown in Figure 12, Oracle's basic memory structure includes: 

The System Global Area (SGA) which contains: 

The Database buffer cache (enabling logical reads) 

The Redo log buffer 
The Shared pool which contains: 

Shared SQL Area where parsed SQL statements are cached for 

shared access by shadow processes 

Row cache which holds the Oracle Data Dictionary information 
Program Global Area (PGA) and Sort Areas 

A user call refers to a shadow process which accesses a shared SQL statement. A recursive call 
refers to the Row Cache making a physical read to load Oracle Data Dictionary objects from the 
system tablespace. To increase the entire shared pool (Row Cache + Shared SQL area), the 
parameter SHARED_POOL_SIZE in INIT.ORA is tuned. 
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Figure 12: Oracle Memory Management for each SAP work Process 

D) Backup and Recovery Management for Oracle in SAP R/3 
Environments 



The type and frequency of backups determine the speed and success of the recovery process. 
Various backup methods exist today. The DBA needs to determine the kind of backup 
procedures that are required for a particular implementation. This section gives an overview of 
various backup types commonly used for SAP R/3 databases. 
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Backups can be broadly categorized into physical backups and logical backups. A physical 
backup is a backup where the actual physical database files are copied from one location to the 
other (usually from disk to tape). Operating system backups, backups using Recovery Manager, 
cold backups, and hot backups are examples of a physical backup. Logical backups use Oracle's 
EXPORT/IMPORT functionality. 

SAP R/3's kernel executables - BRBACKUP and BRRESTORE can be used to manage database 
backups and log file backups. During a database backup, the data files, the online redo log files, 
profiles, and control file are backed up. 



Internal Operation of Hot Backup [10]: 

During an Online backup, the Oracle database and the R/3 System remain available. Oracle 
automatically writes to the next online REDO log file in a round robin fashion while the HOT 
BACKUP is in progress. Owing to continuous Data Manipulation Langurage (DML) activity, 
Online backups as referred to in Section 4A and as detailed below, cause a reduction in system 
performance. 

At the ALTER TABLESPACE BEGIN BACKUP command, the file header's SCN is advanced 
to the SCN number captured when this command is issued. The checkpoint SCN in the backup 
files must be the same as when the backup started. After the initial checkpoint, succeeding 
checkpoints cease to update the file headers when in HOT BACKUP mode. 

When an ALTER TABLESPACE BEGIN BACKUP command is issued, the data files that 
belong to the tablespaces get flagged as HOT-BACKUP-IN-PROGRESS. This command 
checkpoints all the data files that are in HOT BACKUP mode. The checkpointing process that 
runs during the execution of the ALTER TABLESPACE BEGIN BACKUP command flushes all 
the dirty buffers that belong to the data files in the tablespaces and ensures that only blocks that 
are changed during the HOT BACKUP are written to Redo Log file. 
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The above activity also points to the important fact that depending on the I/O activity, excessive 
redo logs can be generated for data files during HOT BACKUP mode as Oracle records the 
activities in the REDO logs. 

The phenomenon of SPLIT BLOCKS arises during the online backup. Depending upon how the 
operating system (AIX) copies blocks, it is possible for a HOT BACKUP to contain an 
inconsistent version of a given data block. This inconsistency arises due to the difference in the 
operating system and Oracle block sizes. By checkpointing and logging the before image of a 
data block to the redo log file before the first change, it can be used later to reconstruct a fractured 
block during recovery. 

To verify the consistency of the block before recovery, Oracle compares the version number at 
the beginning of the block to the version number at the end of the block to determine whether the 
block has been split during a hot backup. The before image of the block in the redo log is copied 
to disk before applying the redo changes. 

In order to make a consistent database copy as mentioned in section 4A, Oracle is put in HOT 
BACKUP mode using the ALTER TABLESPACE BEGIN BACKUP command on all the 
tablespaces on the primary system. In this mode, Oracle does not stop writing to the datafiles (it 
actually allows continued operation of the database almost exactly as during normal operation). 

The database files are read by server processes and written by the database writer (DBWR) 
throughout the backup, as they are when a backup is not taking place. The only difference 
manifested in the open database files is the freezing of the checkpoint SCN, and the incrementing 
of the HOT-BACKUP SCN. 

The following actions take place: 

1. Oracle checkpoints the tablespace, flushing all changes from shared memory to disk 
while the SCN markers for each datafile in that tablespace are "frozen" at their current 
values. 
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2. While further updates will be sent to the datafiles, the SCN markers will not be updated 
until the tablespace is taken out of backup mode. Oracle switches to logging full images 
of changed database blocks to the redo logs. 

3. To avoid inconsistencies in block images as mentioned before about SPLIT BLOCK 
technique, it would log the entire image of the block after the change instead of recording 
the change vector that could show how it changed a particular block. 

4. The SMBR works through this datafile, backing it up at volume / LUN level. After all the 
tablespaces have been put in HOT BACKUP mode, certain CLI commands are executed 
to activate the ESS's advanced functions such as FlashCopy, a near-instant local volume 
copy function. Upon the execution of this command, Oracle's HOT BACKUP mode is 
terminated by the execution of the END BACKUP command, which returns database to 
normal state. 

5. After an online backup is finished, if using the SAP backup management tool 
BRBACKUP, a log switch is initiated This creates a new online redo log file and the 
Oracle Archiver process in turn copies the previously active online redo log file directly 
into /oracle/<SID>/saparch directory. The entire log information generated during the 
online backup is contained in the offline redo log files. 

6. When an attempt to start the SAP/Oracle instance is made on the "decme" host, Oracle 
looks at the data file and sees an old SCN value - the SCN marker it had before the hot 
backup began. When a "recover" command is applied, Oracle begins to apply redo logs 
against this data file. Since the redo logs contain a complete image of every block that 
changed during the backup, Oracle can rebuild this file to a consistent state. 

Upon the execution of a full Online backup, the following files are backed up: 
The data files of all tablespaces belonging to the Oracle database 
The control file 

- The profiles (init<SID>.ora, init<SID>.sap, and init<SID>.dba) 
Archive log files 

Contents of /oracle/SSD/sapreorg, /oracle/SSD/sapbackup directories 
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Logical Backups 

During a logical backup of an Oracle database, using Oracle EXPORT utility, the data in the 
database is copied but the location of the data is not recorded. This utility copies the data and the 
database definitions or the Meta data and saves them to a binary OS file in Oracle internal format. 
While a logical backup takes longer to complete than a physical backup, it is used to detect data 
block corruptions, restore tables accidentally lost by users and reduce database fragmentation. 

Using Oracle's IMPORT utility, the data is restored back from the binary Oracle internal format 
into the Oracle database format. Thus Logical backups provide a very flexible tool to migrate an 
SAP / Oracle database from one machine / platform to another. Logical Backups using EXPORT 
/ IMPORT features are widely used by SAP R/3 to create homogeneous system copies. 

SAP R/3 Backup Objects 

SAP delivers the following tools for performing database backups: 

- The program BRBACKUP backs up the data files, the control file, and the database redo log 
files where necessary. 

- The program BRARCHIVE backs up the offline redo log files of the database. 

- Both BRBACKUP and BRARCHIVE record the actions performed in log files. 

- These log files can be used in case of a database restore, and can be analyzed by the program 
BRRESTORE. This program can restore all files belonging to the database system from the 
backups 

- The database backup tools support standard backups (disk and tape). 

External data management systems like Tivoli Storage Manager's Tivoli Data Protector for R/3 or 
Legato's Networker enable backup and recovery procedures to be integrated with the existing 
operations of an SAP R/3 landscape. These systems improve backup throughput and allow easy 
management of large number of tapes. 
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Overview of Recovery Issues in SAP / Oracle Environments 

Every recovery option for SAP R/3 using Oracle is very important and has its own use, and it is 
crucial that DBA's understand how each recovery option works. Recovery processes vary, 
depending on the type of failure that has occurred, the structures that have been affected, and the 
type of recovery that is required. 

On the recovery side, some recovery procedures require DBA intervention, whereas other internal 
recovery mechanisms are transparent to the DBA. 

Block-level recovery is the simplest type of recovery, and is automatically done by SAP / Oracle. 
It is done when a process dies just as it is changing a buffer. The online redo logs for the current 
thread are used to reconstruct the buffer and then write to disk. If a process dies abnormally while 
modifying a block, SAP / Oracle will do a block-level recovery, which is automatic and doesn't 
require human intervention. 

On the other hand, if a data file has been lost, recovery requires additional steps. Some of the 
common errors or failures include the following [10]: 

- User error 

- Statement failure 

- Process failure 

- Network failure 

- Instance failure 

- Media failure 

A user deleting a row or dropping a table is a typical example of user error. The recovery 
procedure might be as simple as importing from a logical backup, or might involve a more 
complicated procedure such as doing point-in-time recovery from a physical backup (DBq) on 
the backup host, exporting the table, and, finally, importing it into the production database. 
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A statement failure occurs when SAP/Oracle statements are selecting from a table that doesn't 
exist and trying to do an insert resulting in the statement's failure due to unavailable space in the 
table. Recovery from such failures is automatic. Upon detection, SAP / Oracle usually will roll 
back the statement, returning control to the user or user program. The user can simply re-execute 
the statement after correcting the problem. 

A process failure due to abnormal termination of a process caused by SAP / Oracle itself (such 
as when a user performs a *C from SQL*PLUS during non standard SAP loads at Oracle level or 
during admin operations). If the process that is terminated is a user process, a server process, or 
an application process, the Process Monitor (PMON) performs process recovery. PMON cleans 
up the cache and frees up resources that the process was using. 

PMON resets the status of the transaction table in the rollback segment for that transaction, 
releasing the locks or latches acquired by the terminated process. It then removes the process ID 
from the list of active processes. 

PMON doesn't clean up the processes that have been killed by Oracle. During an abnormal 
background process termination, Oracle must be shut down and restarted.Oracle uses SMON to 
automatically perform roll forward, and Oracle's transaction recovery which will roll back any 
uncommitted transactions. 

Network failures, can occur in a SAP three-tier configuration. In such cases, PMON will roll 
back the uncommitted Logical Unit of work (LUW) of each of the processes. 

An instance failure can be caused by a physical (hardware) or a design (software) problem— for 
example, when one of the database background processes (DBWR) detects that there is a problem 
on the non-RAID5 disk and can't write to it. 

In such a situation, an error message is written to a log file and the background process 
terminates. Crash recovery or instance recovery is automatically done depending on the I/O 
activity at the failure time, database instance recovery might take a long time. 
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Crash recovery has to roll forward and then transaction recovery has to roll the transaction back, 
which might take a long period of time. 

Thread recovery is done automatically by SAP / Oracle when it discovers that an instance died 
leaving a thread open. Thread recovery is performed as part of either crash recovery or instance 
recovery. If the database has a single instance, then crash recovery is performed. This requires the 
DBA to simply start up the database, and crash recovery is automatically performed by Oracle. 

If multiple instances are accessing the database and if one of the instances crashes, the second 
instance automatically performs instance recovery to recover the first thread. Either way, the goal 
of thread recovery is to restore the data block changes that were in the cache of the instance that 
died, and to close the thread that was left open. Thread recovery always uses the online redo log 
files of the thread it is recovering. 

Media recovery, the last type of recovery, is executed in response to a recovery command. It is 
used to make a backup data file become current, or to restore changes that were lost when a data 
file went offline without a checkpoint. During media recovery, archived logs as well as online log 
files can be applied. 

In an ESS RAIDS environment these types of failures are minimized owing to the subsystem's 
fault tolerant design. 

Media failures are the most dangerous failures in non-RAID5 environments resulting in potential 
loss of data if proper backup procedures are not followed, and could involve more time to recover 
in comparison with other kinds of failures mentioned above. A typical example of a media failure 
is a non-RAID5 disk controller failure or a disk head crash, which causes all Oracle database files 
residing on that disk (or disks) to be lost. 

However, with ESS's fault tolerant RAIDS architecture, the impact of media failures is greatly 
minimized. 
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SAP R/3 data layout 

SAP R/3 and the importance of Database Layout in Oracle Environments [7]: 

With the progressive complexity in application system designs, combined with rapid explosion in 
database sizes, it is necessary to distribute data in disk systems in a way that allows the fastest 
possible access to it. While only 1% of the tablespaces in the SAP R/3 system are characterized 
by high growth and thus high I/O impact, these tablespaces essentially dictate the overall size and 
performance of the database. Storage architectures that provide the capability to distribute these 
tablespaces across as many disks as possible are able to deliver superior performance in this 
demanding OLTP environment These architectures include the ability to reassign very high 
growth objects to new tablespaces. 

In preparation for SAP R/3 Oracle data layout for the ESS, the following sequence of activities 
needs to be considered: 

- Find Number and types of SAP systems to be installed in the system landscape that 
typically includes Development, Quality Assurance and Productive system instances. 

- Start with the Planning of an SAP landscape topology with estimation of hardware 
requirements for storage, network and server machines. This creates a hardware 
requirements list: ESS storage subsystems, number of SCSI / FC connections, server 
machines, network connectivity etc. 

- Sizing done by HW partners to establish requirements for: 

- CPU 

Storage space, bandwidth 

- Network bandwidth 

- System infrastructure / landscape (inter system communication) 
SAP installation guide 

Assignment of SAP systems to hosts 

Estimations for storage needs of individual database systems / tablespaces 
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Collect space requirements from SAP applications, their databases (tablespaces, 
logging), 3 rd party software (Bolt-ons) 

Consider ESS / Host / DB / Application restrictions 

Map requirements to AIX logical volumes 

Group the logical volumes to volume groups 

Design ESS configuration and perform configuration steps with ESS Specialist 
(creating LUN's as shown in Figure 16) to create AIX volume groups and logical 
volumes 

SAP R/3 Backup Objects and File Definitions for Oracle 

All objects in the R/3 environment need to be backed up. These objects are: 

1. R/3 data objects: 

a. R/3 archiving objects 

b. R/3 Interfaces 

c. SAP Executables 

2. Computing center data such as: 

a. The Operating System 

b. Third Party Programs connected to R/3 

3. Database objects 

Figure 13 depicts the SAP Backup Objects and Figure 4 depicts the Oracle File Systems for SAP. 
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Oracle Backup Objects for SAP 



Computing Center Data 





Figure 13: Oracle Backup Objects for SAP 



As depicted in Figure 13, SAP R/3 database backup is accomplished through the SAPDBA tool 
that invokes SAP's BRBACKUP, BRRESTORE and BRARCHIVE utilities. These utilities, in 
turn, are integrated into TSM using SAP approved BACKINT interface also called TDP for R/3. 
Computing Center data and R/3 data are backed up using Operating System utilities (cpio, tar) or 
by using products such as TSM, Legato Networker, and Veritas Net Backup etc. 

As shown in Oracle File System for SAP in Figure 4, the volume groups LSSlOdata, LSS12data, 
LSSHdata and LSS16data are 64GB each in size. The other volume groups are sapvg comprising 
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of 1GB LUNs as shown in Figure 7. The volume group sapvg is used for the SAP and Oracle 
binaries, and logvg holds the online and archive logs. 

SAP R/3 uses standardized Directory and File names in the Oracle environment. As shown in 
Figure 13, the SAP R/3 objects - Oracle control files, Oracle data files, offline redo log files & 
online redo log files are required for a consistent database backup and restore/recovery. 

The Oracle binaries are located in the subdirectory 'bin 1 of the ORACLEHOME directory. The 
ORACLEHOME directory is also required on each server with a database client. The 
environment variables SAPDATA HOME and SAPDATA<1, 2, 3 ...6> point to the directories 
containing database-specific files, such as data files, online redo log files, and offline redo log 
files. The operating system users <SlD>adm (System ID) and ora<SID> (on the database server) 
require the following environment variable: 

ORACLE_SID = <SID> (System ID on the database server, SSD in Figure 4) 

E) ESS RaidS Disk Layout Considerations for SAP R/3 Environments 

The RAIDS arrays / ranks are connected via device adapters to the ESS fault tolerant control unit 
that contains, cache and NVRAM as shown in the Figure 14 below. The control unit attaches to 
host systems via SCSI or fiber channel adapters. 

With proper distribution of all kinds of data across all available devices using state of the art 
storage systems like ESS, the whole I/O subsystem is made available to each individual database 
and file objects (tablespace, directory, table, index). 

As hotspots tend to occur on a subset of the tablespaces or even on a single tablespace, by striping 
these tablespaces across all devices, the cumulative performance of all devices and device 
adapters is made available to each individual tablespace to improve performance. 
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ESS's RAIDS architecture, with maximum width striping of all Oracle tablespaces, relieves the 
DBA from hotspot identification and tracking which might arise from changing application 
profiles, shifting workloads and rapid data growths in typical non-RAID5 storage environments. 

The RAID5 striping of ESS accommodates single tablespace growth while totally avoiding any 
additional cost of striping tablespaces across all devices. 
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Figure 14: ESS's Fault Tolerant Architecture and Logical Sub Systems Design 



Each Oracle tablespace and file system is striped across all RAID5 volumes. This is the optimum 
configuration since the ESS's built-in I/O capabilities are automatically and dynamically 
"allocated" on demand to all the tablespaces. 
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Spreading a logical volume across separate hdisks on the same array would provide no 
performance benefit, because all of the activity is still going to a single RAID5 array. The best 
technique for maximizing throughput of ESS is to balance workload across as many RAIDS 
arrays as possible. 

ESS's configuration for performance may need to be balanced with availability considerations. If 
all databases sharing the ESS are required to be highly available, then the databases will have to 
be spread over the ESS clusters and Device Adapter (DA) pairs to ensure that there are alternate 
paths to the same data. This may lead to data files of multiple databases residing on the same 
array/s. Such configurations should be laid out with due attention to the I/O load that a single 
array can handle and possible I/O contention. Keeping in mind that it is the array that determines 
the throughput, not the number of logical disks, consideration for LUN sizes for the ESS should 
be based on the application and customer requirements. For handling high sequential read/write 
I/O as in Business Warehouse workloads, ensure that the tablespace layout makes use of all the 
arrays in an ESS. 

Typically, one would allocate 8 or 16 GB LUNs over the available storage and then distribute the 
data across the RAIDS arrays to get optimum performance from the ESS. Further distribution of 
I/O is achieved by creating AIX logical volumes using the ESS logical volumes defined in 
multiple RAID arrays. Monitoring the I/O access pattern can then be used for further tuning and 
re-distributing heavily accessed database files across the arrays. 



F) Sub System Device Driver (SDD) 

ESS's SDD, a High Availability Automatic I/O Path Failover Function in the ESS, enhances data 
availability. It takes advantage of multiple active paths, distributes the I/O workload and supports 
more than one path from the host to the shared LUNs. If a failure occurs in the data path between 
the Host System and the ESS, SDD automatically switches the I/O to another path. It will also 
move the failed path back online after a repair is made. A single LUN can appear as 2 to 
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16 LUNs. This avoids I/O bottlenecks that could occur when too many I/O operations are directed 
to a common device using the same I/O path. 

The following Figure 15 highlights the necessary steps in preparing the layout for an ESS based 
SAP R/3 Oracle system using fixed block (FB) for Open systems format. 
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Figure 15: Steps in preparing the ESS for data layout and connectivity to hosts 

In AIX, the Logical Volume Manager uses the 8GB LUNs, defined in ESS, by assigning them to 
volume groups as hdisks. Each ESS logical disk translates to AIX like an hdisk as shown in 
Figure 16. With each hdisk in AIX, the host limits the number of concurrent requests to a value of 
'20' that it will send to ESS. 
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Figure 16: Vpath to Hdisk Mapping by SDD 
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G) Volume Layout for Oracle 

The following table, Figure 17, captures the volume layout for SAP R/3 in ORACLE / ESS 
environment with a mapping of vpaths to hdisks using Subsystem Device Driver (SDD). 
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Figure 17: ESS Volume Mapping & Layout 
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H) The Role of the ESS Specialist 

With ESS Specialist, the WebGUI interface for the Enterprise Server, all the tasks needed for 
setting up arrays, logical subsystems, logical unit numberings, host definitions for connectivity 
via SCSI / Fiber, PPRC path definitions and Copy services tasks can be administered and 
managed. This is shown in Figure 18. 
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Figure 18: ESS Specialist Views and Functions 
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